Note: This is test shard 2 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from RebalanceIgnoredTserversTest
[ RUN ] RebalanceIgnoredTserversTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 20:08:32.866941 30573 test_util.cc:268] Using random seed: 218630724
I20240501 20:08:32.878592 30573 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:08:32.878700 30573 ts_itest-base.cc:113] --------------
I20240501 20:08:32.878749 30573 ts_itest-base.cc:114] 5 tablet servers
I20240501 20:08:32.878813 30573 ts_itest-base.cc:115] 3 replicas per TS
I20240501 20:08:32.878881 30573 ts_itest-base.cc:116] --------------
2024-05-01T20:08:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:08:32Z Disabled control of system clock
I20240501 20:08:32.905447 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.219.126:35289
--webserver_interface=127.29.219.126
--webserver_port=0
--builtin_ntp_servers=127.29.219.84:33275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.219.126:35289
--enable_range_replica_placement=false with env {}
W20240501 20:08:33.242123 30585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:33.242441 30585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:33.242583 30585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:33.250306 30585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:08:33.250448 30585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:33.250502 30585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:08:33.250598 30585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:08:33.259932 30585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:33275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.29.219.126:35289
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.219.126:35289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.29.219.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:33.261350 30585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:33.263180 30585 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:33.274415 30590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:33.275512 30593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:33.276544 30591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:33.277395 30585 server_base.cc:1034] running on GCE node
I20240501 20:08:33.278275 30585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:33.279857 30585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:33.281118 30585 hybrid_clock.cc:648] HybridClock initialized: now 1714594113281049 us; error 43 us; skew 500 ppm
I20240501 20:08:33.281462 30585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:33.284036 30585 webserver.cc:460] Webserver started at http://127.29.219.126:35409/ using document root <none> and password file <none>
I20240501 20:08:33.284780 30585 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:33.284914 30585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:33.285216 30585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:33.288018 30585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "2ce3dc39668340cfb91e87ab189a20ee"
format_stamp: "Formatted at 2024-05-01 20:08:33 on dist-test-slave-h2cj"
I20240501 20:08:33.288789 30585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "2ce3dc39668340cfb91e87ab189a20ee"
format_stamp: "Formatted at 2024-05-01 20:08:33 on dist-test-slave-h2cj"
I20240501 20:08:33.294484 30585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 20:08:33.298066 30599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:33.299445 30585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:08:33.299616 30585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2ce3dc39668340cfb91e87ab189a20ee"
format_stamp: "Formatted at 2024-05-01 20:08:33 on dist-test-slave-h2cj"
I20240501 20:08:33.299803 30585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:33.330107 30585 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:33.331178 30585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:33.331527 30585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:33.351143 30585 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.126:35289
I20240501 20:08:33.351166 30650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.126:35289 every 8 connection(s)
I20240501 20:08:33.352715 30585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:08:33.353847 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 30585
I20240501 20:08:33.354187 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:08:33.357085 30651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:33.368616 30651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ce3dc39668340cfb91e87ab189a20ee: Bootstrap starting.
I20240501 20:08:33.373299 30651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ce3dc39668340cfb91e87ab189a20ee: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:33.374591 30651 log.cc:824] T 00000000000000000000000000000000 P 2ce3dc39668340cfb91e87ab189a20ee: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:33.378011 30651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ce3dc39668340cfb91e87ab189a20ee: No bootstrap required, opened a new log
I20240501 20:08:33.386386 30651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ce3dc39668340cfb91e87ab189a20ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce3dc39668340cfb91e87ab189a20ee" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 35289 } }
I20240501 20:08:33.386804 30651 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ce3dc39668340cfb91e87ab189a20ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:33.386931 30651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ce3dc39668340cfb91e87ab189a20ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ce3dc39668340cfb91e87ab189a20ee, State: Initialized, Role: FOLLOWER
I20240501 20:08:33.387579 30651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ce3dc39668340cfb91e87ab189a20ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce3dc39668340cfb91e87ab189a20ee" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 35289 } }
I20240501 20:08:33.387770 30651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ce3dc39668340cfb91e87ab189a20ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:33.387899 30651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ce3dc39668340cfb91e87ab189a20ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:33.388051 30651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ce3dc39668340cfb91e87ab189a20ee [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:33.390441 30651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ce3dc39668340cfb91e87ab189a20ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce3dc39668340cfb91e87ab189a20ee" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 35289 } }
I20240501 20:08:33.391100 30651 leader_election.cc:302] T 00000000000000000000000000000000 P 2ce3dc39668340cfb91e87ab189a20ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ce3dc39668340cfb91e87ab189a20ee; no voters:
I20240501 20:08:33.391692 30651 leader_election.cc:288] T 00000000000000000000000000000000 P 2ce3dc39668340cfb91e87ab189a20ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:33.391844 30656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ce3dc39668340cfb91e87ab189a20ee [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:33.392470 30656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ce3dc39668340cfb91e87ab189a20ee [term 1 LEADER]: Becoming Leader. State: Replica: 2ce3dc39668340cfb91e87ab189a20ee, State: Running, Role: LEADER
I20240501 20:08:33.393049 30656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ce3dc39668340cfb91e87ab189a20ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce3dc39668340cfb91e87ab189a20ee" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 35289 } }
I20240501 20:08:33.393787 30651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ce3dc39668340cfb91e87ab189a20ee [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:08:33.396286 30657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ce3dc39668340cfb91e87ab189a20ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ce3dc39668340cfb91e87ab189a20ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce3dc39668340cfb91e87ab189a20ee" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 35289 } } }
I20240501 20:08:33.396286 30658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ce3dc39668340cfb91e87ab189a20ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ce3dc39668340cfb91e87ab189a20ee. Latest consensus state: current_term: 1 leader_uuid: "2ce3dc39668340cfb91e87ab189a20ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ce3dc39668340cfb91e87ab189a20ee" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 35289 } } }
I20240501 20:08:33.396700 30657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ce3dc39668340cfb91e87ab189a20ee [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:33.396751 30658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ce3dc39668340cfb91e87ab189a20ee [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:33.397503 30661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:08:33.403546 30661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:08:33.412623 30661 catalog_manager.cc:1347] Generated new cluster ID: bae543b24eb84b0a8d744cfdb379d2b5
I20240501 20:08:33.412791 30661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:08:33.436487 30661 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:08:33.437745 30661 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:08:33.451969 30661 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2ce3dc39668340cfb91e87ab189a20ee: Generated new TSK 0
I20240501 20:08:33.452934 30661 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:08:33.462580 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.65:0
--local_ip_for_outbound_sockets=127.29.219.65
--webserver_interface=127.29.219.65
--webserver_port=0
--tserver_master_addrs=127.29.219.126:35289
--builtin_ntp_servers=127.29.219.84:33275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240501 20:08:33.811058 30675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:33.811359 30675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:33.811456 30675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:33.819061 30675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:33.819254 30675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.65
I20240501 20:08:33.828570 30675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:33275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.29.219.65
--webserver_port=0
--tserver_master_addrs=127.29.219.126:35289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.65
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:33.829985 30675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:33.831720 30675 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:33.843638 30680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:33.844493 30681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:33.845624 30683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:33.846506 30675 server_base.cc:1034] running on GCE node
I20240501 20:08:33.847200 30675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:33.848253 30675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:33.849514 30675 hybrid_clock.cc:648] HybridClock initialized: now 1714594113849427 us; error 49 us; skew 500 ppm
I20240501 20:08:33.849913 30675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:33.856979 30675 webserver.cc:460] Webserver started at http://127.29.219.65:41791/ using document root <none> and password file <none>
I20240501 20:08:33.857796 30675 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:33.857926 30675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:33.858223 30675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:33.861238 30675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "79df09093d5e4bf481ca1213493d954d"
format_stamp: "Formatted at 2024-05-01 20:08:33 on dist-test-slave-h2cj"
I20240501 20:08:33.862073 30675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "79df09093d5e4bf481ca1213493d954d"
format_stamp: "Formatted at 2024-05-01 20:08:33 on dist-test-slave-h2cj"
I20240501 20:08:33.867869 30675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20240501 20:08:33.871702 30689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:33.873167 30675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:08:33.873364 30675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "79df09093d5e4bf481ca1213493d954d"
format_stamp: "Formatted at 2024-05-01 20:08:33 on dist-test-slave-h2cj"
I20240501 20:08:33.873569 30675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:33.892702 30675 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:33.893604 30675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:33.893940 30675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:33.895414 30675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:33.897866 30675 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:33.897980 30675 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:33.898128 30675 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:33.898200 30675 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:33.933619 30675 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.65:46007
I20240501 20:08:33.933681 30801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.65:46007 every 8 connection(s)
I20240501 20:08:33.935072 30675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:08:33.942059 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 30675
I20240501 20:08:33.942270 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:08:33.947204 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.66:0
--local_ip_for_outbound_sockets=127.29.219.66
--webserver_interface=127.29.219.66
--webserver_port=0
--tserver_master_addrs=127.29.219.126:35289
--builtin_ntp_servers=127.29.219.84:33275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 20:08:33.949935 30802 heartbeater.cc:347] Connected to a master server at 127.29.219.126:35289
I20240501 20:08:33.950309 30802 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:33.951542 30802 heartbeater.cc:511] Master 127.29.219.126:35289 requested a full tablet report, sending...
I20240501 20:08:33.953929 30616 ts_manager.cc:194] Registered new tserver with Master: 79df09093d5e4bf481ca1213493d954d (127.29.219.65:46007)
I20240501 20:08:33.955509 30616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.65:52639
W20240501 20:08:34.279083 30806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:34.279438 30806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:34.279578 30806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:34.287460 30806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:34.287701 30806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.66
I20240501 20:08:34.296887 30806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:33275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.29.219.66
--webserver_port=0
--tserver_master_addrs=127.29.219.126:35289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.66
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:34.298254 30806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:34.300025 30806 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:34.312613 30811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:34.313486 30812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:34.314391 30814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:34.315236 30806 server_base.cc:1034] running on GCE node
I20240501 20:08:34.315876 30806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:34.316792 30806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:34.318039 30806 hybrid_clock.cc:648] HybridClock initialized: now 1714594114317961 us; error 52 us; skew 500 ppm
I20240501 20:08:34.318348 30806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:34.320829 30806 webserver.cc:460] Webserver started at http://127.29.219.66:45803/ using document root <none> and password file <none>
I20240501 20:08:34.321692 30806 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:34.321835 30806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:34.322142 30806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:34.325008 30806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e22a514136494780984ebb6c16bd1ec1"
format_stamp: "Formatted at 2024-05-01 20:08:34 on dist-test-slave-h2cj"
I20240501 20:08:34.325814 30806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e22a514136494780984ebb6c16bd1ec1"
format_stamp: "Formatted at 2024-05-01 20:08:34 on dist-test-slave-h2cj"
I20240501 20:08:34.331192 30806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 20:08:34.334841 30820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:34.336380 30806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:08:34.336560 30806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e22a514136494780984ebb6c16bd1ec1"
format_stamp: "Formatted at 2024-05-01 20:08:34 on dist-test-slave-h2cj"
I20240501 20:08:34.336786 30806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:34.350916 30806 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:34.351842 30806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:34.352191 30806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:34.353339 30806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:34.354919 30806 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:34.355003 30806 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:34.355139 30806 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:34.355202 30806 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:34.389045 30806 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.66:37333
I20240501 20:08:34.389081 30932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.66:37333 every 8 connection(s)
I20240501 20:08:34.390350 30806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:08:34.396195 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 30806
I20240501 20:08:34.396369 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:08:34.400059 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.67:0
--local_ip_for_outbound_sockets=127.29.219.67
--webserver_interface=127.29.219.67
--webserver_port=0
--tserver_master_addrs=127.29.219.126:35289
--builtin_ntp_servers=127.29.219.84:33275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 20:08:34.404844 30933 heartbeater.cc:347] Connected to a master server at 127.29.219.126:35289
I20240501 20:08:34.405246 30933 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:34.406363 30933 heartbeater.cc:511] Master 127.29.219.126:35289 requested a full tablet report, sending...
I20240501 20:08:34.407999 30616 ts_manager.cc:194] Registered new tserver with Master: e22a514136494780984ebb6c16bd1ec1 (127.29.219.66:37333)
I20240501 20:08:34.408816 30616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.66:55037
W20240501 20:08:34.734659 30936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:34.734992 30936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:34.735126 30936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:34.742837 30936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:34.743062 30936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.67
I20240501 20:08:34.752274 30936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:33275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.29.219.67
--webserver_port=0
--tserver_master_addrs=127.29.219.126:35289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.67
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:34.753641 30936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:34.755460 30936 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:34.767588 30944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:34.768589 30942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:34.769516 30941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:34.770421 30936 server_base.cc:1034] running on GCE node
I20240501 20:08:34.771095 30936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:34.772063 30936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:34.773260 30936 hybrid_clock.cc:648] HybridClock initialized: now 1714594114773189 us; error 47 us; skew 500 ppm
I20240501 20:08:34.773545 30936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:34.775787 30936 webserver.cc:460] Webserver started at http://127.29.219.67:42937/ using document root <none> and password file <none>
I20240501 20:08:34.776494 30936 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:34.776602 30936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:34.776859 30936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:34.779634 30936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "349eae50d44943fa986a813f3d065720"
format_stamp: "Formatted at 2024-05-01 20:08:34 on dist-test-slave-h2cj"
I20240501 20:08:34.780371 30936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "349eae50d44943fa986a813f3d065720"
format_stamp: "Formatted at 2024-05-01 20:08:34 on dist-test-slave-h2cj"
I20240501 20:08:34.785501 30936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 20:08:34.788807 30950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:34.790131 30936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:08:34.790283 30936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "349eae50d44943fa986a813f3d065720"
format_stamp: "Formatted at 2024-05-01 20:08:34 on dist-test-slave-h2cj"
I20240501 20:08:34.790439 30936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:34.812984 30936 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:34.813803 30936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:34.814064 30936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:34.815012 30936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:34.816450 30936 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:34.816529 30936 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:34.816618 30936 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:34.816663 30936 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:34.848402 30936 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.67:45569
I20240501 20:08:34.848438 31062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.67:45569 every 8 connection(s)
I20240501 20:08:34.849704 30936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:08:34.850389 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 30936
I20240501 20:08:34.850602 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:08:34.853947 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.68:0
--local_ip_for_outbound_sockets=127.29.219.68
--webserver_interface=127.29.219.68
--webserver_port=0
--tserver_master_addrs=127.29.219.126:35289
--builtin_ntp_servers=127.29.219.84:33275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 20:08:34.864348 31063 heartbeater.cc:347] Connected to a master server at 127.29.219.126:35289
I20240501 20:08:34.864709 31063 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:34.865541 31063 heartbeater.cc:511] Master 127.29.219.126:35289 requested a full tablet report, sending...
I20240501 20:08:34.866997 30616 ts_manager.cc:194] Registered new tserver with Master: 349eae50d44943fa986a813f3d065720 (127.29.219.67:45569)
I20240501 20:08:34.867771 30616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.67:58481
I20240501 20:08:34.958480 30802 heartbeater.cc:503] Master 127.29.219.126:35289 was elected leader, sending a full tablet report...
W20240501 20:08:35.193017 31066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:35.193341 31066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:35.193509 31066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:35.201375 31066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:35.201629 31066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.68
I20240501 20:08:35.210971 31066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:33275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.29.219.68
--webserver_port=0
--tserver_master_addrs=127.29.219.126:35289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.68
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:35.212361 31066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:35.214190 31066 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:35.226531 31071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:35.227411 31074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:35.228377 31072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:35.229194 31066 server_base.cc:1034] running on GCE node
I20240501 20:08:35.229823 31066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:35.230803 31066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:35.232025 31066 hybrid_clock.cc:648] HybridClock initialized: now 1714594115231964 us; error 40 us; skew 500 ppm
I20240501 20:08:35.232332 31066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:35.234660 31066 webserver.cc:460] Webserver started at http://127.29.219.68:35225/ using document root <none> and password file <none>
I20240501 20:08:35.235406 31066 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:35.235538 31066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:35.235842 31066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:35.238559 31066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "a2b454feaa354bc6ab58ee2bb246eaeb"
format_stamp: "Formatted at 2024-05-01 20:08:35 on dist-test-slave-h2cj"
I20240501 20:08:35.239405 31066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "a2b454feaa354bc6ab58ee2bb246eaeb"
format_stamp: "Formatted at 2024-05-01 20:08:35 on dist-test-slave-h2cj"
I20240501 20:08:35.244881 31066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20240501 20:08:35.248386 31080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:35.249758 31066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 20:08:35.249917 31066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "a2b454feaa354bc6ab58ee2bb246eaeb"
format_stamp: "Formatted at 2024-05-01 20:08:35 on dist-test-slave-h2cj"
I20240501 20:08:35.250097 31066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:35.267603 31066 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:35.268463 31066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:35.268774 31066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:35.270304 31066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:35.271889 31066 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:35.271968 31066 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:35.272104 31066 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:35.272157 31066 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:35.304657 31066 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.68:34385
I20240501 20:08:35.304704 31192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.68:34385 every 8 connection(s)
I20240501 20:08:35.306002 31066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:08:35.313540 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 31066
I20240501 20:08:35.313735 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:08:35.317005 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.69:0
--local_ip_for_outbound_sockets=127.29.219.69
--webserver_interface=127.29.219.69
--webserver_port=0
--tserver_master_addrs=127.29.219.126:35289
--builtin_ntp_servers=127.29.219.84:33275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 20:08:35.319584 31193 heartbeater.cc:347] Connected to a master server at 127.29.219.126:35289
I20240501 20:08:35.319932 31193 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:35.320830 31193 heartbeater.cc:511] Master 127.29.219.126:35289 requested a full tablet report, sending...
I20240501 20:08:35.322530 30616 ts_manager.cc:194] Registered new tserver with Master: a2b454feaa354bc6ab58ee2bb246eaeb (127.29.219.68:34385)
I20240501 20:08:35.323398 30616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.68:54425
I20240501 20:08:35.411957 30933 heartbeater.cc:503] Master 127.29.219.126:35289 was elected leader, sending a full tablet report...
W20240501 20:08:35.654767 31196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:35.655097 31196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:35.655267 31196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:35.662770 31196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:35.662993 31196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.69
I20240501 20:08:35.671947 31196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:33275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.29.219.69
--webserver_port=0
--tserver_master_addrs=127.29.219.126:35289
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.69
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:35.673358 31196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:35.675165 31196 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:35.686936 31204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:35.687852 31202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:35.688738 31201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:35.689693 31196 server_base.cc:1034] running on GCE node
I20240501 20:08:35.690352 31196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:35.691416 31196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:35.692637 31196 hybrid_clock.cc:648] HybridClock initialized: now 1714594115692572 us; error 37 us; skew 500 ppm
I20240501 20:08:35.692975 31196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:35.695519 31196 webserver.cc:460] Webserver started at http://127.29.219.69:38235/ using document root <none> and password file <none>
I20240501 20:08:35.696286 31196 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:35.696410 31196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:35.696710 31196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:35.704918 31196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "7213e704252343b1bc37691e0a08e2dc"
format_stamp: "Formatted at 2024-05-01 20:08:35 on dist-test-slave-h2cj"
I20240501 20:08:35.705790 31196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "7213e704252343b1bc37691e0a08e2dc"
format_stamp: "Formatted at 2024-05-01 20:08:35 on dist-test-slave-h2cj"
I20240501 20:08:35.711499 31196 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240501 20:08:35.714952 31210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:35.716361 31196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 20:08:35.716524 31196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "7213e704252343b1bc37691e0a08e2dc"
format_stamp: "Formatted at 2024-05-01 20:08:35 on dist-test-slave-h2cj"
I20240501 20:08:35.716714 31196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:35.740571 31196 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:35.741461 31196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:35.741787 31196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:35.742862 31196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:35.744333 31196 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:35.744403 31196 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:35.744531 31196 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:35.744580 31196 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:35.776316 31196 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.69:36751
I20240501 20:08:35.776369 31322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.69:36751 every 8 connection(s)
I20240501 20:08:35.777724 31196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:08:35.785985 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 31196
I20240501 20:08:35.786177 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 20:08:35.790361 31323 heartbeater.cc:347] Connected to a master server at 127.29.219.126:35289
I20240501 20:08:35.790693 31323 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:35.791473 31323 heartbeater.cc:511] Master 127.29.219.126:35289 requested a full tablet report, sending...
I20240501 20:08:35.792773 30615 ts_manager.cc:194] Registered new tserver with Master: 7213e704252343b1bc37691e0a08e2dc (127.29.219.69:36751)
I20240501 20:08:35.793623 30615 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.69:56691
I20240501 20:08:35.802512 30573 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 20:08:35.824676 30615 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38776:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 20:08:35.858060 31128 tablet_service.cc:1467] Processing CreateTablet for tablet c38d2fe34dc04acb8bcc7fdc8f97ca7e (DEFAULT_TABLE table=TestTable [id=0d7bd35969954cee9e9c6340169ffb8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:35.859707 31128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c38d2fe34dc04acb8bcc7fdc8f97ca7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:35.863498 31258 tablet_service.cc:1467] Processing CreateTablet for tablet c38d2fe34dc04acb8bcc7fdc8f97ca7e (DEFAULT_TABLE table=TestTable [id=0d7bd35969954cee9e9c6340169ffb8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:35.865082 31258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c38d2fe34dc04acb8bcc7fdc8f97ca7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:35.866211 30737 tablet_service.cc:1467] Processing CreateTablet for tablet c38d2fe34dc04acb8bcc7fdc8f97ca7e (DEFAULT_TABLE table=TestTable [id=0d7bd35969954cee9e9c6340169ffb8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:35.867511 30737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c38d2fe34dc04acb8bcc7fdc8f97ca7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:35.870276 31063 heartbeater.cc:503] Master 127.29.219.126:35289 was elected leader, sending a full tablet report...
I20240501 20:08:35.874013 31341 tablet_bootstrap.cc:492] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P a2b454feaa354bc6ab58ee2bb246eaeb: Bootstrap starting.
I20240501 20:08:35.878283 31342 tablet_bootstrap.cc:492] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc: Bootstrap starting.
I20240501 20:08:35.879130 31341 tablet_bootstrap.cc:654] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P a2b454feaa354bc6ab58ee2bb246eaeb: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:35.880183 31343 tablet_bootstrap.cc:492] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d: Bootstrap starting.
I20240501 20:08:35.880462 31341 log.cc:824] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P a2b454feaa354bc6ab58ee2bb246eaeb: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:35.881986 31342 tablet_bootstrap.cc:654] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:35.883157 31341 tablet_bootstrap.cc:492] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P a2b454feaa354bc6ab58ee2bb246eaeb: No bootstrap required, opened a new log
I20240501 20:08:35.883337 31342 log.cc:824] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:35.883404 31341 ts_tablet_manager.cc:1367] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P a2b454feaa354bc6ab58ee2bb246eaeb: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240501 20:08:35.883878 31343 tablet_bootstrap.cc:654] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:35.885079 31343 log.cc:824] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:35.886696 31342 tablet_bootstrap.cc:492] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc: No bootstrap required, opened a new log
I20240501 20:08:35.886973 31342 ts_tablet_manager.cc:1367] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240501 20:08:35.887388 31343 tablet_bootstrap.cc:492] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d: No bootstrap required, opened a new log
I20240501 20:08:35.887614 31343 ts_tablet_manager.cc:1367] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240501 20:08:35.892555 31341 raft_consensus.cc:348] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P a2b454feaa354bc6ab58ee2bb246eaeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } }
I20240501 20:08:35.893041 31341 raft_consensus.cc:374] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P a2b454feaa354bc6ab58ee2bb246eaeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:35.893146 31341 raft_consensus.cc:729] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P a2b454feaa354bc6ab58ee2bb246eaeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2b454feaa354bc6ab58ee2bb246eaeb, State: Initialized, Role: FOLLOWER
I20240501 20:08:35.893293 31343 raft_consensus.cc:348] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } }
I20240501 20:08:35.893731 31343 raft_consensus.cc:374] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:35.893831 31343 raft_consensus.cc:729] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79df09093d5e4bf481ca1213493d954d, State: Initialized, Role: FOLLOWER
I20240501 20:08:35.893929 31341 consensus_queue.cc:260] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P a2b454feaa354bc6ab58ee2bb246eaeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } }
I20240501 20:08:35.895244 31343 consensus_queue.cc:260] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } }
I20240501 20:08:35.896066 31342 raft_consensus.cc:348] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } }
I20240501 20:08:35.896543 31342 raft_consensus.cc:374] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:35.896659 31342 raft_consensus.cc:729] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7213e704252343b1bc37691e0a08e2dc, State: Initialized, Role: FOLLOWER
I20240501 20:08:35.897536 31342 consensus_queue.cc:260] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } }
I20240501 20:08:35.903597 31343 ts_tablet_manager.cc:1398] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20240501 20:08:35.905804 31193 heartbeater.cc:503] Master 127.29.219.126:35289 was elected leader, sending a full tablet report...
I20240501 20:08:35.905890 31323 heartbeater.cc:503] Master 127.29.219.126:35289 was elected leader, sending a full tablet report...
I20240501 20:08:35.906461 31341 ts_tablet_manager.cc:1398] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P a2b454feaa354bc6ab58ee2bb246eaeb: Time spent starting tablet: real 0.023s user 0.014s sys 0.009s
I20240501 20:08:35.907616 31342 ts_tablet_manager.cc:1398] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc: Time spent starting tablet: real 0.021s user 0.017s sys 0.001s
W20240501 20:08:35.938330 30803 tablet.cc:2369] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:35.951884 31349 raft_consensus.cc:482] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:35.952159 31349 raft_consensus.cc:504] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } }
I20240501 20:08:35.954017 31349 leader_election.cc:288] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2b454feaa354bc6ab58ee2bb246eaeb (127.29.219.68:34385), 79df09093d5e4bf481ca1213493d954d (127.29.219.65:46007)
I20240501 20:08:35.957021 31348 raft_consensus.cc:482] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:35.957286 31348 raft_consensus.cc:504] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } }
I20240501 20:08:35.959280 31348 leader_election.cc:288] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2b454feaa354bc6ab58ee2bb246eaeb (127.29.219.68:34385), 7213e704252343b1bc37691e0a08e2dc (127.29.219.69:36751)
I20240501 20:08:35.963088 31148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38d2fe34dc04acb8bcc7fdc8f97ca7e" candidate_uuid: "7213e704252343b1bc37691e0a08e2dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" is_pre_election: true
I20240501 20:08:35.963723 31148 raft_consensus.cc:2454] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P a2b454feaa354bc6ab58ee2bb246eaeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7213e704252343b1bc37691e0a08e2dc in term 0.
I20240501 20:08:35.964491 31214 leader_election.cc:302] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7213e704252343b1bc37691e0a08e2dc, a2b454feaa354bc6ab58ee2bb246eaeb; no voters:
I20240501 20:08:35.965279 31349 raft_consensus.cc:2781] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:35.965514 31349 raft_consensus.cc:482] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:35.965698 31349 raft_consensus.cc:3037] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:35.967097 30757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38d2fe34dc04acb8bcc7fdc8f97ca7e" candidate_uuid: "7213e704252343b1bc37691e0a08e2dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79df09093d5e4bf481ca1213493d954d" is_pre_election: true
I20240501 20:08:35.967579 30757 raft_consensus.cc:2454] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7213e704252343b1bc37691e0a08e2dc in term 0.
I20240501 20:08:35.968238 31349 raft_consensus.cc:504] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } }
I20240501 20:08:35.968811 31349 leader_election.cc:288] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [CANDIDATE]: Term 1 election: Requested vote from peers a2b454feaa354bc6ab58ee2bb246eaeb (127.29.219.68:34385), 79df09093d5e4bf481ca1213493d954d (127.29.219.65:46007)
I20240501 20:08:35.969313 31148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38d2fe34dc04acb8bcc7fdc8f97ca7e" candidate_uuid: "7213e704252343b1bc37691e0a08e2dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb"
I20240501 20:08:35.969499 30757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38d2fe34dc04acb8bcc7fdc8f97ca7e" candidate_uuid: "7213e704252343b1bc37691e0a08e2dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79df09093d5e4bf481ca1213493d954d"
I20240501 20:08:35.969514 31148 raft_consensus.cc:3037] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P a2b454feaa354bc6ab58ee2bb246eaeb [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:35.969691 30757 raft_consensus.cc:3037] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:35.971980 31148 raft_consensus.cc:2454] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P a2b454feaa354bc6ab58ee2bb246eaeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7213e704252343b1bc37691e0a08e2dc in term 1.
I20240501 20:08:35.972370 31214 leader_election.cc:302] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7213e704252343b1bc37691e0a08e2dc, a2b454feaa354bc6ab58ee2bb246eaeb; no voters:
I20240501 20:08:35.972642 30757 raft_consensus.cc:2454] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7213e704252343b1bc37691e0a08e2dc in term 1.
I20240501 20:08:35.972844 31349 raft_consensus.cc:2781] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:35.975190 31349 raft_consensus.cc:686] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [term 1 LEADER]: Becoming Leader. State: Replica: 7213e704252343b1bc37691e0a08e2dc, State: Running, Role: LEADER
I20240501 20:08:35.975564 31148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38d2fe34dc04acb8bcc7fdc8f97ca7e" candidate_uuid: "79df09093d5e4bf481ca1213493d954d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" is_pre_election: true
I20240501 20:08:35.975868 31148 raft_consensus.cc:2379] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P a2b454feaa354bc6ab58ee2bb246eaeb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 79df09093d5e4bf481ca1213493d954d in current term 1: Already voted for candidate 7213e704252343b1bc37691e0a08e2dc in this term.
I20240501 20:08:35.975939 31349 consensus_queue.cc:237] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } }
I20240501 20:08:35.976418 31278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38d2fe34dc04acb8bcc7fdc8f97ca7e" candidate_uuid: "79df09093d5e4bf481ca1213493d954d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7213e704252343b1bc37691e0a08e2dc" is_pre_election: true
I20240501 20:08:35.978055 30693 leader_election.cc:302] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79df09093d5e4bf481ca1213493d954d; no voters: 7213e704252343b1bc37691e0a08e2dc, a2b454feaa354bc6ab58ee2bb246eaeb
I20240501 20:08:35.978492 31348 raft_consensus.cc:2726] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:08:35.981078 30615 catalog_manager.cc:5484] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 7213e704252343b1bc37691e0a08e2dc (127.29.219.69). New cstate: current_term: 1 leader_uuid: "7213e704252343b1bc37691e0a08e2dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:35.994390 30573 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 20:08:35.997103 30573 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 79df09093d5e4bf481ca1213493d954d to finish bootstrapping
I20240501 20:08:36.006145 30573 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e22a514136494780984ebb6c16bd1ec1 to finish bootstrapping
I20240501 20:08:36.015200 30573 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 349eae50d44943fa986a813f3d065720 to finish bootstrapping
I20240501 20:08:36.023885 30573 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a2b454feaa354bc6ab58ee2bb246eaeb to finish bootstrapping
W20240501 20:08:36.029613 31324 tablet.cc:2369] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:36.031502 30573 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7213e704252343b1bc37691e0a08e2dc to finish bootstrapping
W20240501 20:08:36.058022 31194 tablet.cc:2369] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P a2b454feaa354bc6ab58ee2bb246eaeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:36.383937 31349 consensus_queue.cc:1035] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:36.393306 31349 consensus_queue.cc:1035] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:36.531334 31258 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:36.535264 30998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:36.546236 30737 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:36.548758 30868 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:36.549010 31128 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:37.180001 31258 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:37.189424 30868 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:37.192396 30998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:37.193687 31128 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:37.206938 30737 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:37.370015 30998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:37.383787 31258 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:37.386099 31128 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:37.403318 30737 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:37.404853 30868 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:37.574828 31258 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:37.578150 30998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:37.581806 30737 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:37.588526 30868 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:37.605309 31128 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:38.137025 30616 ts_manager.cc:292] Set tserver state for 79df09093d5e4bf481ca1213493d954d to MAINTENANCE_MODE
I20240501 20:08:39.685171 30998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:39.685827 31258 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:39.689069 30868 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:39.693800 31128 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:39.695293 30737 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:39.869787 31258 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:39.872053 30737 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:39.874173 30998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:39.884011 30868 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:39.893388 31128 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:40.074084 30998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:40.076180 31258 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:40.088940 30737 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:40.089272 31128 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:40.090765 30868 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:40.157768 31278 consensus_queue.cc:237] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } attrs { replace: true } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } }
I20240501 20:08:40.160127 30757 raft_consensus.cc:1261] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d [term 1 FOLLOWER]: Refusing update from remote peer 7213e704252343b1bc37691e0a08e2dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:08:40.160235 31148 raft_consensus.cc:1261] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P a2b454feaa354bc6ab58ee2bb246eaeb [term 1 FOLLOWER]: Refusing update from remote peer 7213e704252343b1bc37691e0a08e2dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:08:40.160750 31400 consensus_queue.cc:1035] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:08:40.161136 31629 consensus_queue.cc:1035] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:08:40.164291 31400 raft_consensus.cc:2932] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } attrs { replace: true } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } } }
I20240501 20:08:40.164786 31148 raft_consensus.cc:2932] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P a2b454feaa354bc6ab58ee2bb246eaeb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } attrs { replace: true } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } } }
I20240501 20:08:40.165160 30757 raft_consensus.cc:2932] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } attrs { replace: true } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } } }
I20240501 20:08:40.170290 30616 catalog_manager.cc:5484] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc reported cstate change: config changed from index -1 to 2. New cstate: current_term: 1 leader_uuid: "7213e704252343b1bc37691e0a08e2dc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:40.179205 31278 consensus_queue.cc:237] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } attrs { replace: true } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } } peers { permanent_uuid: "e22a514136494780984ebb6c16bd1ec1" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 37333 } attrs { promote: true } }
I20240501 20:08:40.182469 31148 raft_consensus.cc:1261] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P a2b454feaa354bc6ab58ee2bb246eaeb [term 1 FOLLOWER]: Refusing update from remote peer 7213e704252343b1bc37691e0a08e2dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:08:40.183053 31629 consensus_queue.cc:1035] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:08:40.183948 30757 raft_consensus.cc:1261] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d [term 1 FOLLOWER]: Refusing update from remote peer 7213e704252343b1bc37691e0a08e2dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:08:40.185045 31629 consensus_queue.cc:1035] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:08:40.185938 31632 raft_consensus.cc:2932] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER e22a514136494780984ebb6c16bd1ec1 (127.29.219.66) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } attrs { replace: true } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } } peers { permanent_uuid: "e22a514136494780984ebb6c16bd1ec1" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 37333 } attrs { promote: true } } }
I20240501 20:08:40.186599 31148 raft_consensus.cc:2932] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P a2b454feaa354bc6ab58ee2bb246eaeb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER e22a514136494780984ebb6c16bd1ec1 (127.29.219.66) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } attrs { replace: true } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } } peers { permanent_uuid: "e22a514136494780984ebb6c16bd1ec1" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 37333 } attrs { promote: true } } }
I20240501 20:08:40.191732 30602 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c38d2fe34dc04acb8bcc7fdc8f97ca7e with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240501 20:08:40.192178 31212 consensus_peers.cc:485] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc -> Peer e22a514136494780984ebb6c16bd1ec1 (127.29.219.66:37333): Couldn't send request to peer e22a514136494780984ebb6c16bd1ec1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c38d2fe34dc04acb8bcc7fdc8f97ca7e. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:40.192730 30614 catalog_manager.cc:5484] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P a2b454feaa354bc6ab58ee2bb246eaeb reported cstate change: config changed from index 2 to 3, NON_VOTER e22a514136494780984ebb6c16bd1ec1 (127.29.219.66) added. New cstate: current_term: 1 leader_uuid: "7213e704252343b1bc37691e0a08e2dc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } attrs { replace: true } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } } peers { permanent_uuid: "e22a514136494780984ebb6c16bd1ec1" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 37333 } attrs { promote: true } } }
I20240501 20:08:40.193511 30757 raft_consensus.cc:2932] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER e22a514136494780984ebb6c16bd1ec1 (127.29.219.66) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } attrs { replace: true } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } } peers { permanent_uuid: "e22a514136494780984ebb6c16bd1ec1" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 37333 } attrs { promote: true } } }
I20240501 20:08:40.691303 31655 ts_tablet_manager.cc:902] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P e22a514136494780984ebb6c16bd1ec1: Initiating tablet copy from peer 7213e704252343b1bc37691e0a08e2dc (127.29.219.69:36751)
I20240501 20:08:40.693018 31655 tablet_copy_client.cc:308] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P e22a514136494780984ebb6c16bd1ec1: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.69:36751
I20240501 20:08:40.699991 31298 tablet_copy_service.cc:140] P 7213e704252343b1bc37691e0a08e2dc: Received BeginTabletCopySession request for tablet c38d2fe34dc04acb8bcc7fdc8f97ca7e from peer e22a514136494780984ebb6c16bd1ec1 ({username='slave'} at 127.29.219.66:45001)
I20240501 20:08:40.700209 31298 tablet_copy_service.cc:161] P 7213e704252343b1bc37691e0a08e2dc: Beginning new tablet copy session on tablet c38d2fe34dc04acb8bcc7fdc8f97ca7e from peer e22a514136494780984ebb6c16bd1ec1 at {username='slave'} at 127.29.219.66:45001: session id = e22a514136494780984ebb6c16bd1ec1-c38d2fe34dc04acb8bcc7fdc8f97ca7e
I20240501 20:08:40.702919 31298 tablet_copy_source_session.cc:205] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:08:40.707625 31655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c38d2fe34dc04acb8bcc7fdc8f97ca7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:40.717587 31655 tablet_copy_client.cc:783] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P e22a514136494780984ebb6c16bd1ec1: tablet copy: Starting download of 0 data blocks...
I20240501 20:08:40.717928 31655 tablet_copy_client.cc:647] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P e22a514136494780984ebb6c16bd1ec1: tablet copy: Starting download of 1 WAL segments...
I20240501 20:08:40.720268 31655 tablet_copy_client.cc:523] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P e22a514136494780984ebb6c16bd1ec1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:08:40.723724 31655 tablet_bootstrap.cc:492] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P e22a514136494780984ebb6c16bd1ec1: Bootstrap starting.
I20240501 20:08:40.730593 31655 log.cc:824] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P e22a514136494780984ebb6c16bd1ec1: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:40.734556 31655 tablet_bootstrap.cc:492] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P e22a514136494780984ebb6c16bd1ec1: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:08:40.735049 31655 tablet_bootstrap.cc:492] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P e22a514136494780984ebb6c16bd1ec1: Bootstrap complete.
I20240501 20:08:40.735440 31655 ts_tablet_manager.cc:1367] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P e22a514136494780984ebb6c16bd1ec1: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.008s
I20240501 20:08:40.741989 31655 raft_consensus.cc:348] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P e22a514136494780984ebb6c16bd1ec1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } attrs { replace: true } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } } peers { permanent_uuid: "e22a514136494780984ebb6c16bd1ec1" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 37333 } attrs { promote: true } }
I20240501 20:08:40.742484 31655 raft_consensus.cc:729] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P e22a514136494780984ebb6c16bd1ec1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e22a514136494780984ebb6c16bd1ec1, State: Initialized, Role: LEARNER
I20240501 20:08:40.743021 31655 consensus_queue.cc:260] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P e22a514136494780984ebb6c16bd1ec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } attrs { replace: true } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } } peers { permanent_uuid: "e22a514136494780984ebb6c16bd1ec1" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 37333 } attrs { promote: true } }
I20240501 20:08:40.744642 31655 ts_tablet_manager.cc:1398] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P e22a514136494780984ebb6c16bd1ec1: Time spent starting tablet: real 0.009s user 0.008s sys 0.001s
I20240501 20:08:40.745688 31298 tablet_copy_service.cc:342] P 7213e704252343b1bc37691e0a08e2dc: Request end of tablet copy session e22a514136494780984ebb6c16bd1ec1-c38d2fe34dc04acb8bcc7fdc8f97ca7e received from {username='slave'} at 127.29.219.66:45001
I20240501 20:08:40.745841 31298 tablet_copy_service.cc:433] P 7213e704252343b1bc37691e0a08e2dc: ending tablet copy session e22a514136494780984ebb6c16bd1ec1-c38d2fe34dc04acb8bcc7fdc8f97ca7e on tablet c38d2fe34dc04acb8bcc7fdc8f97ca7e with peer e22a514136494780984ebb6c16bd1ec1
I20240501 20:08:41.117473 30888 raft_consensus.cc:1203] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P e22a514136494780984ebb6c16bd1ec1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20240501 20:08:41.665606 31674 raft_consensus.cc:1050] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc: attempting to promote NON_VOTER e22a514136494780984ebb6c16bd1ec1 to VOTER
I20240501 20:08:41.666353 31674 consensus_queue.cc:237] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } attrs { replace: true } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } } peers { permanent_uuid: "e22a514136494780984ebb6c16bd1ec1" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 37333 } attrs { promote: false } }
I20240501 20:08:41.669018 30888 raft_consensus.cc:1261] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P e22a514136494780984ebb6c16bd1ec1 [term 1 LEARNER]: Refusing update from remote peer 7213e704252343b1bc37691e0a08e2dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:08:41.669273 30757 raft_consensus.cc:1261] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d [term 1 FOLLOWER]: Refusing update from remote peer 7213e704252343b1bc37691e0a08e2dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:08:41.669528 31148 raft_consensus.cc:1261] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P a2b454feaa354bc6ab58ee2bb246eaeb [term 1 FOLLOWER]: Refusing update from remote peer 7213e704252343b1bc37691e0a08e2dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:08:41.669695 31674 consensus_queue.cc:1035] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e22a514136494780984ebb6c16bd1ec1" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 37333 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:08:41.670658 31690 consensus_queue.cc:1035] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 20:08:41.670955 31692 consensus_queue.cc:1035] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:08:41.674423 31674 raft_consensus.cc:2932] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, e22a514136494780984ebb6c16bd1ec1 (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } attrs { replace: true } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } } peers { permanent_uuid: "e22a514136494780984ebb6c16bd1ec1" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 37333 } attrs { promote: false } } }
I20240501 20:08:41.675307 31148 raft_consensus.cc:2932] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P a2b454feaa354bc6ab58ee2bb246eaeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, e22a514136494780984ebb6c16bd1ec1 (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } attrs { replace: true } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } } peers { permanent_uuid: "e22a514136494780984ebb6c16bd1ec1" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 37333 } attrs { promote: false } } }
I20240501 20:08:41.678579 30757 raft_consensus.cc:2932] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, e22a514136494780984ebb6c16bd1ec1 (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } attrs { replace: true } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } } peers { permanent_uuid: "e22a514136494780984ebb6c16bd1ec1" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 37333 } attrs { promote: false } } }
I20240501 20:08:41.681924 30615 catalog_manager.cc:5484] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc reported cstate change: config changed from index 3 to 4, e22a514136494780984ebb6c16bd1ec1 (127.29.219.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7213e704252343b1bc37691e0a08e2dc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e22a514136494780984ebb6c16bd1ec1" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 37333 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:41.691045 30888 raft_consensus.cc:2932] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P e22a514136494780984ebb6c16bd1ec1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, e22a514136494780984ebb6c16bd1ec1 (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "79df09093d5e4bf481ca1213493d954d" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 46007 } attrs { replace: true } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } } peers { permanent_uuid: "e22a514136494780984ebb6c16bd1ec1" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 37333 } attrs { promote: false } } }
I20240501 20:08:41.693176 31278 consensus_queue.cc:237] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } } peers { permanent_uuid: "e22a514136494780984ebb6c16bd1ec1" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 37333 } attrs { promote: false } }
I20240501 20:08:41.696228 31148 raft_consensus.cc:1261] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P a2b454feaa354bc6ab58ee2bb246eaeb [term 1 FOLLOWER]: Refusing update from remote peer 7213e704252343b1bc37691e0a08e2dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:08:41.696952 31693 consensus_queue.cc:1035] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:08:41.699529 31690 raft_consensus.cc:2932] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 79df09093d5e4bf481ca1213493d954d (127.29.219.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } } peers { permanent_uuid: "e22a514136494780984ebb6c16bd1ec1" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 37333 } attrs { promote: false } } }
I20240501 20:08:41.700387 31148 raft_consensus.cc:2932] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P a2b454feaa354bc6ab58ee2bb246eaeb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 79df09093d5e4bf481ca1213493d954d (127.29.219.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } } peers { permanent_uuid: "e22a514136494780984ebb6c16bd1ec1" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 37333 } attrs { promote: false } } }
I20240501 20:08:41.701414 30887 raft_consensus.cc:1261] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P e22a514136494780984ebb6c16bd1ec1 [term 1 FOLLOWER]: Refusing update from remote peer 7213e704252343b1bc37691e0a08e2dc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:08:41.702782 31693 consensus_queue.cc:1035] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e22a514136494780984ebb6c16bd1ec1" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 37333 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:08:41.703439 30602 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c38d2fe34dc04acb8bcc7fdc8f97ca7e with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:08:41.704556 30888 raft_consensus.cc:2932] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P e22a514136494780984ebb6c16bd1ec1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 79df09093d5e4bf481ca1213493d954d (127.29.219.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } } peers { permanent_uuid: "e22a514136494780984ebb6c16bd1ec1" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 37333 } attrs { promote: false } } }
I20240501 20:08:41.705278 30616 catalog_manager.cc:5484] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc reported cstate change: config changed from index 4 to 5, VOTER 79df09093d5e4bf481ca1213493d954d (127.29.219.65) evicted. New cstate: current_term: 1 leader_uuid: "7213e704252343b1bc37691e0a08e2dc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a2b454feaa354bc6ab58ee2bb246eaeb" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 34385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7213e704252343b1bc37691e0a08e2dc" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 36751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e22a514136494780984ebb6c16bd1ec1" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 37333 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:41.709590 30737 tablet_service.cc:1514] Processing DeleteTablet for tablet c38d2fe34dc04acb8bcc7fdc8f97ca7e with delete_type TABLET_DATA_TOMBSTONED (TS 79df09093d5e4bf481ca1213493d954d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35590
I20240501 20:08:41.710496 31704 tablet_replica.cc:330] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d: stopping tablet replica
I20240501 20:08:41.711050 31704 raft_consensus.cc:2229] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:08:41.711496 31704 raft_consensus.cc:2258] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:08:41.713066 31704 ts_tablet_manager.cc:1875] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:08:41.717819 31704 ts_tablet_manager.cc:1888] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:08:41.718446 31704 log.cc:1196] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 79df09093d5e4bf481ca1213493d954d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/c38d2fe34dc04acb8bcc7fdc8f97ca7e
I20240501 20:08:41.719403 30603 catalog_manager.cc:4830] TS 79df09093d5e4bf481ca1213493d954d (127.29.219.65:46007): tablet c38d2fe34dc04acb8bcc7fdc8f97ca7e (table TestTable [id=0d7bd35969954cee9e9c6340169ffb8a]) successfully deleted
I20240501 20:08:42.157738 30998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:42.162770 31128 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:42.168530 31258 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:42.170199 30737 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:42.179738 30868 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:42.337731 30998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:42.339154 31258 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:42.340101 30737 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:42.345611 31128 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:42.355648 30868 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:42.521432 30737 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:42.522863 31258 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:42.525252 30998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:42.538440 30868 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:42.539342 31128 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:42.702807 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 30806
I20240501 20:08:42.704546 30927 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:42.812490 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 30806
W20240501 20:08:42.832326 30602 connection.cc:535] server connection from 127.29.219.66:55037 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:08:43.019001 31212 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.219.66:37333: connect: Connection refused (error 111)
W20240501 20:08:43.020437 31212 consensus_peers.cc:485] T c38d2fe34dc04acb8bcc7fdc8f97ca7e P 7213e704252343b1bc37691e0a08e2dc -> Peer e22a514136494780984ebb6c16bd1ec1 (127.29.219.66:37333): Couldn't send request to peer e22a514136494780984ebb6c16bd1ec1. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.66:37333: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:43.343963 30998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:43.344947 31258 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:43.346374 30737 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:43.347468 31128 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:43.492956 30998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:43.493340 31258 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:43.500535 30737 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:43.501042 31128 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:43.678896 31258 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:43.679327 30998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:43.680281 30737 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:43.696708 31128 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:43.841351 31258 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:43.852059 30737 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:43.852461 31128 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:43.853498 30998 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:44.021732 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 30675
I20240501 20:08:44.023093 30797 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:44.141052 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 30675
I20240501 20:08:44.165875 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 30936
I20240501 20:08:44.167093 31057 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:44.276947 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 30936
I20240501 20:08:44.297541 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 31066
I20240501 20:08:44.298669 31188 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:44.415105 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 31066
I20240501 20:08:44.436700 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 31196
I20240501 20:08:44.437816 31317 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:44.559839 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 31196
W20240501 20:08:44.582226 30602 connection.cc:535] server connection from 127.29.219.69:56691 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:08:44.582392 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 30585
I20240501 20:08:44.583648 30646 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:44.720732 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 30585
2024-05-01T20:08:44Z chronyd exiting
[ OK ] RebalanceIgnoredTserversTest.Basic (11905 ms)
[----------] 1 test from RebalanceIgnoredTserversTest (11905 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/1
I20240501 20:08:44.769806 30573 test_util.cc:268] Using random seed: 230533607
I20240501 20:08:44.771901 30573 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:08:44.771994 30573 ts_itest-base.cc:113] --------------
I20240501 20:08:44.772025 30573 ts_itest-base.cc:114] 5 tablet servers
I20240501 20:08:44.772075 30573 ts_itest-base.cc:115] 3 replicas per TS
I20240501 20:08:44.772114 30573 ts_itest-base.cc:116] --------------
2024-05-01T20:08:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:08:44Z Disabled control of system clock
I20240501 20:08:44.797561 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.219.126:39585
--webserver_interface=127.29.219.126
--webserver_port=0
--builtin_ntp_servers=127.29.219.84:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.219.126:39585
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 20:08:45.131892 31941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:45.132220 31941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:45.132455 31941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:45.140493 31941 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:45.140642 31941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:08:45.140698 31941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:45.140787 31941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:08:45.140853 31941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:08:45.150638 31941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.29.219.126:39585
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.219.126:39585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.29.219.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:45.152074 31941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:45.153888 31941 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:45.165397 31947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:45.166306 31946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:45.167291 31949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:45.168362 31941 server_base.cc:1034] running on GCE node
I20240501 20:08:45.169246 31941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:45.170886 31941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:45.172112 31941 hybrid_clock.cc:648] HybridClock initialized: now 1714594125172058 us; error 46 us; skew 500 ppm
I20240501 20:08:45.172473 31941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:45.180929 31941 webserver.cc:460] Webserver started at http://127.29.219.126:45689/ using document root <none> and password file <none>
I20240501 20:08:45.181779 31941 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:45.181905 31941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:45.182255 31941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:45.185111 31941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ba10d1506b644d6385119e6889a6833f"
format_stamp: "Formatted at 2024-05-01 20:08:45 on dist-test-slave-h2cj"
I20240501 20:08:45.185956 31941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ba10d1506b644d6385119e6889a6833f"
format_stamp: "Formatted at 2024-05-01 20:08:45 on dist-test-slave-h2cj"
I20240501 20:08:45.191526 31941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240501 20:08:45.195091 31955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:45.196609 31941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240501 20:08:45.196838 31941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ba10d1506b644d6385119e6889a6833f"
format_stamp: "Formatted at 2024-05-01 20:08:45 on dist-test-slave-h2cj"
I20240501 20:08:45.197044 31941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:45.221959 31941 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:45.222894 31941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:45.223181 31941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:45.241845 31941 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.126:39585
I20240501 20:08:45.241869 32006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.126:39585 every 8 connection(s)
I20240501 20:08:45.243292 31941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:08:45.246372 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 31941
I20240501 20:08:45.246594 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:08:45.247220 32007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:45.259361 32007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba10d1506b644d6385119e6889a6833f: Bootstrap starting.
I20240501 20:08:45.262835 32007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba10d1506b644d6385119e6889a6833f: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:45.263805 32007 log.cc:824] T 00000000000000000000000000000000 P ba10d1506b644d6385119e6889a6833f: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:45.266548 32007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba10d1506b644d6385119e6889a6833f: No bootstrap required, opened a new log
I20240501 20:08:45.273460 32007 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba10d1506b644d6385119e6889a6833f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba10d1506b644d6385119e6889a6833f" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 39585 } }
I20240501 20:08:45.273854 32007 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba10d1506b644d6385119e6889a6833f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:45.273963 32007 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba10d1506b644d6385119e6889a6833f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba10d1506b644d6385119e6889a6833f, State: Initialized, Role: FOLLOWER
I20240501 20:08:45.274577 32007 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba10d1506b644d6385119e6889a6833f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba10d1506b644d6385119e6889a6833f" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 39585 } }
I20240501 20:08:45.274827 32007 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba10d1506b644d6385119e6889a6833f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:45.274957 32007 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba10d1506b644d6385119e6889a6833f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:45.275112 32007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba10d1506b644d6385119e6889a6833f [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:45.277189 32007 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba10d1506b644d6385119e6889a6833f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba10d1506b644d6385119e6889a6833f" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 39585 } }
I20240501 20:08:45.277695 32007 leader_election.cc:302] T 00000000000000000000000000000000 P ba10d1506b644d6385119e6889a6833f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba10d1506b644d6385119e6889a6833f; no voters:
I20240501 20:08:45.278295 32007 leader_election.cc:288] T 00000000000000000000000000000000 P ba10d1506b644d6385119e6889a6833f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:45.278429 32012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba10d1506b644d6385119e6889a6833f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:45.279392 32012 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba10d1506b644d6385119e6889a6833f [term 1 LEADER]: Becoming Leader. State: Replica: ba10d1506b644d6385119e6889a6833f, State: Running, Role: LEADER
I20240501 20:08:45.280198 32012 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba10d1506b644d6385119e6889a6833f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba10d1506b644d6385119e6889a6833f" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 39585 } }
I20240501 20:08:45.280409 32007 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba10d1506b644d6385119e6889a6833f [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:08:45.283308 32013 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba10d1506b644d6385119e6889a6833f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba10d1506b644d6385119e6889a6833f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba10d1506b644d6385119e6889a6833f" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 39585 } } }
I20240501 20:08:45.283705 32013 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba10d1506b644d6385119e6889a6833f [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:45.283288 32014 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba10d1506b644d6385119e6889a6833f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba10d1506b644d6385119e6889a6833f. Latest consensus state: current_term: 1 leader_uuid: "ba10d1506b644d6385119e6889a6833f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba10d1506b644d6385119e6889a6833f" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 39585 } } }
I20240501 20:08:45.284269 32014 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba10d1506b644d6385119e6889a6833f [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:45.287858 32019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:08:45.293087 32019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:08:45.301450 32019 catalog_manager.cc:1347] Generated new cluster ID: 4b257922a67845559d37564736e69998
I20240501 20:08:45.301607 32019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:08:45.315485 32019 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:08:45.316980 32019 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:08:45.326485 32019 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ba10d1506b644d6385119e6889a6833f: Generated new TSK 0
I20240501 20:08:45.327471 32019 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:08:45.344581 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.65:0
--local_ip_for_outbound_sockets=127.29.219.65
--webserver_interface=127.29.219.65
--webserver_port=0
--tserver_master_addrs=127.29.219.126:39585
--builtin_ntp_servers=127.29.219.84:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:08:45.686463 32031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:45.686877 32031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:45.687079 32031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:45.695104 32031 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:45.695237 32031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:45.695408 32031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.65
I20240501 20:08:45.705121 32031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.29.219.65
--webserver_port=0
--tserver_master_addrs=127.29.219.126:39585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.65
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:45.706519 32031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:45.708362 32031 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:45.721797 32039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:45.722694 32037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:45.723443 32036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:45.724566 32031 server_base.cc:1034] running on GCE node
I20240501 20:08:45.725225 32031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:45.726219 32031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:45.727447 32031 hybrid_clock.cc:648] HybridClock initialized: now 1714594125727382 us; error 59 us; skew 500 ppm
I20240501 20:08:45.727797 32031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:45.730293 32031 webserver.cc:460] Webserver started at http://127.29.219.65:44487/ using document root <none> and password file <none>
I20240501 20:08:45.731165 32031 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:45.731298 32031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:45.731671 32031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:45.734872 32031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5a10add746b347f3a8a654539df2bb79"
format_stamp: "Formatted at 2024-05-01 20:08:45 on dist-test-slave-h2cj"
I20240501 20:08:45.735706 32031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5a10add746b347f3a8a654539df2bb79"
format_stamp: "Formatted at 2024-05-01 20:08:45 on dist-test-slave-h2cj"
I20240501 20:08:45.741122 32031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:08:45.744983 32045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:45.746500 32031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:08:45.746706 32031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5a10add746b347f3a8a654539df2bb79"
format_stamp: "Formatted at 2024-05-01 20:08:45 on dist-test-slave-h2cj"
I20240501 20:08:45.746922 32031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:45.775187 32031 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:45.776103 32031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:45.776381 32031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:45.777385 32031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:45.778995 32031 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:45.779095 32031 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:45.779213 32031 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:45.779286 32031 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:45.811563 32031 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.65:35361
I20240501 20:08:45.811645 32157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.65:35361 every 8 connection(s)
I20240501 20:08:45.812939 32031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:08:45.813577 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 32031
I20240501 20:08:45.813784 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:08:45.817385 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.66:0
--local_ip_for_outbound_sockets=127.29.219.66
--webserver_interface=127.29.219.66
--webserver_port=0
--tserver_master_addrs=127.29.219.126:39585
--builtin_ntp_servers=127.29.219.84:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:45.826987 32158 heartbeater.cc:347] Connected to a master server at 127.29.219.126:39585
I20240501 20:08:45.827359 32158 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:45.828447 32158 heartbeater.cc:511] Master 127.29.219.126:39585 requested a full tablet report, sending...
I20240501 20:08:45.831202 31972 ts_manager.cc:194] Registered new tserver with Master: 5a10add746b347f3a8a654539df2bb79 (127.29.219.65:35361)
I20240501 20:08:45.833719 31972 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.65:59735
W20240501 20:08:46.161942 32162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:46.162269 32162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:46.162358 32162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:46.170099 32162 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:46.170250 32162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:46.170387 32162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.66
I20240501 20:08:46.179800 32162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.29.219.66
--webserver_port=0
--tserver_master_addrs=127.29.219.126:39585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.66
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:46.181321 32162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:46.183238 32162 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:46.195267 32168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:46.196173 32170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:46.197207 32167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:46.198146 32162 server_base.cc:1034] running on GCE node
I20240501 20:08:46.198911 32162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:46.199934 32162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:46.201193 32162 hybrid_clock.cc:648] HybridClock initialized: now 1714594126201126 us; error 43 us; skew 500 ppm
I20240501 20:08:46.201526 32162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:46.203941 32162 webserver.cc:460] Webserver started at http://127.29.219.66:44929/ using document root <none> and password file <none>
I20240501 20:08:46.204681 32162 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:46.204802 32162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:46.205132 32162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:46.207813 32162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "afbbb4b93f0e4ae2a43473307178a5bb"
format_stamp: "Formatted at 2024-05-01 20:08:46 on dist-test-slave-h2cj"
I20240501 20:08:46.208566 32162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "afbbb4b93f0e4ae2a43473307178a5bb"
format_stamp: "Formatted at 2024-05-01 20:08:46 on dist-test-slave-h2cj"
I20240501 20:08:46.214026 32162 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 20:08:46.217612 32176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:46.219034 32162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:08:46.219198 32162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "afbbb4b93f0e4ae2a43473307178a5bb"
format_stamp: "Formatted at 2024-05-01 20:08:46 on dist-test-slave-h2cj"
I20240501 20:08:46.219394 32162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:46.232149 32162 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:46.232978 32162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:46.233244 32162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:46.234172 32162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:46.235733 32162 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:46.235831 32162 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:46.235951 32162 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:46.236024 32162 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:46.270134 32162 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.66:42887
I20240501 20:08:46.270171 32288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.66:42887 every 8 connection(s)
I20240501 20:08:46.271622 32162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:08:46.277458 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 32162
I20240501 20:08:46.277683 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:08:46.281100 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.67:0
--local_ip_for_outbound_sockets=127.29.219.67
--webserver_interface=127.29.219.67
--webserver_port=0
--tserver_master_addrs=127.29.219.126:39585
--builtin_ntp_servers=127.29.219.84:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:46.285674 32289 heartbeater.cc:347] Connected to a master server at 127.29.219.126:39585
I20240501 20:08:46.286010 32289 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:46.286886 32289 heartbeater.cc:511] Master 127.29.219.126:39585 requested a full tablet report, sending...
I20240501 20:08:46.288506 31972 ts_manager.cc:194] Registered new tserver with Master: afbbb4b93f0e4ae2a43473307178a5bb (127.29.219.66:42887)
I20240501 20:08:46.289672 31972 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.66:49629
W20240501 20:08:46.613692 32292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:46.613998 32292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:46.614084 32292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:46.621620 32292 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:46.621760 32292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:46.621876 32292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.67
I20240501 20:08:46.631078 32292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.29.219.67
--webserver_port=0
--tserver_master_addrs=127.29.219.126:39585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.67
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:46.632359 32292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:46.634023 32292 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:46.646374 32298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:46.647105 32297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:46.648027 32300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:46.648453 32292 server_base.cc:1034] running on GCE node
I20240501 20:08:46.649127 32292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:46.650132 32292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:46.651378 32292 hybrid_clock.cc:648] HybridClock initialized: now 1714594126651327 us; error 43 us; skew 500 ppm
I20240501 20:08:46.651752 32292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:46.654165 32292 webserver.cc:460] Webserver started at http://127.29.219.67:39303/ using document root <none> and password file <none>
I20240501 20:08:46.655009 32292 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:46.655129 32292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:46.655462 32292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:46.658109 32292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a1cf2ef277f34512925b234044ea6521"
format_stamp: "Formatted at 2024-05-01 20:08:46 on dist-test-slave-h2cj"
I20240501 20:08:46.658943 32292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a1cf2ef277f34512925b234044ea6521"
format_stamp: "Formatted at 2024-05-01 20:08:46 on dist-test-slave-h2cj"
I20240501 20:08:46.664247 32292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 20:08:46.667853 32306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:46.669265 32292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:08:46.669425 32292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a1cf2ef277f34512925b234044ea6521"
format_stamp: "Formatted at 2024-05-01 20:08:46 on dist-test-slave-h2cj"
I20240501 20:08:46.669656 32292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:46.691910 32292 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:46.692873 32292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:46.693173 32292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:46.694216 32292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:46.695971 32292 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:46.696082 32292 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:46.696223 32292 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:46.696305 32292 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:46.728024 32292 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.67:37795
I20240501 20:08:46.728078 32418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.67:37795 every 8 connection(s)
I20240501 20:08:46.729635 32292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:08:46.731401 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 32292
I20240501 20:08:46.731567 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:08:46.734972 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.68:0
--local_ip_for_outbound_sockets=127.29.219.68
--webserver_interface=127.29.219.68
--webserver_port=0
--tserver_master_addrs=127.29.219.126:39585
--builtin_ntp_servers=127.29.219.84:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:46.746865 32419 heartbeater.cc:347] Connected to a master server at 127.29.219.126:39585
I20240501 20:08:46.747200 32419 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:46.748137 32419 heartbeater.cc:511] Master 127.29.219.126:39585 requested a full tablet report, sending...
I20240501 20:08:46.749842 31972 ts_manager.cc:194] Registered new tserver with Master: a1cf2ef277f34512925b234044ea6521 (127.29.219.67:37795)
I20240501 20:08:46.751224 31972 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.67:49619
I20240501 20:08:46.838173 32158 heartbeater.cc:503] Master 127.29.219.126:39585 was elected leader, sending a full tablet report...
W20240501 20:08:47.075583 32422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:47.075892 32422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:47.076014 32422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:47.083762 32422 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:47.083896 32422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:47.084055 32422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.68
I20240501 20:08:47.093497 32422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.29.219.68
--webserver_port=0
--tserver_master_addrs=127.29.219.126:39585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.68
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:47.094952 32422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:47.096731 32422 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:47.109678 32430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:47.109930 32428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:47.110925 32427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:47.111892 32422 server_base.cc:1034] running on GCE node
I20240501 20:08:47.112555 32422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:47.113524 32422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:47.114717 32422 hybrid_clock.cc:648] HybridClock initialized: now 1714594127114648 us; error 44 us; skew 500 ppm
I20240501 20:08:47.115036 32422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:47.117358 32422 webserver.cc:460] Webserver started at http://127.29.219.68:41853/ using document root <none> and password file <none>
I20240501 20:08:47.118108 32422 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:47.118206 32422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:47.118539 32422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:47.121194 32422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "7362bc6774b844a7b81931533fca8258"
format_stamp: "Formatted at 2024-05-01 20:08:47 on dist-test-slave-h2cj"
I20240501 20:08:47.121987 32422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "7362bc6774b844a7b81931533fca8258"
format_stamp: "Formatted at 2024-05-01 20:08:47 on dist-test-slave-h2cj"
I20240501 20:08:47.127384 32422 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 20:08:47.130903 32436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:47.132414 32422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:08:47.132589 32422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "7362bc6774b844a7b81931533fca8258"
format_stamp: "Formatted at 2024-05-01 20:08:47 on dist-test-slave-h2cj"
I20240501 20:08:47.132855 32422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:47.161159 32422 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:47.162106 32422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:47.162391 32422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:47.163401 32422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:47.164899 32422 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:47.164985 32422 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:47.165089 32422 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:47.165158 32422 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:47.196952 32422 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.68:37329
I20240501 20:08:47.197005 32548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.68:37329 every 8 connection(s)
I20240501 20:08:47.198239 32422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:08:47.206557 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 32422
I20240501 20:08:47.206794 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:08:47.210417 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.69:0
--local_ip_for_outbound_sockets=127.29.219.69
--webserver_interface=127.29.219.69
--webserver_port=0
--tserver_master_addrs=127.29.219.126:39585
--builtin_ntp_servers=127.29.219.84:38841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:47.211619 32549 heartbeater.cc:347] Connected to a master server at 127.29.219.126:39585
I20240501 20:08:47.212123 32549 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:47.213289 32549 heartbeater.cc:511] Master 127.29.219.126:39585 requested a full tablet report, sending...
I20240501 20:08:47.214991 31972 ts_manager.cc:194] Registered new tserver with Master: 7362bc6774b844a7b81931533fca8258 (127.29.219.68:37329)
I20240501 20:08:47.215808 31972 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.68:47131
I20240501 20:08:47.292377 32289 heartbeater.cc:503] Master 127.29.219.126:39585 was elected leader, sending a full tablet report...
W20240501 20:08:47.549968 32552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:47.550309 32552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:47.550462 32552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:47.558115 32552 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:47.558279 32552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:47.558449 32552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.69
I20240501 20:08:47.567711 32552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:38841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.29.219.69
--webserver_port=0
--tserver_master_addrs=127.29.219.126:39585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.69
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:47.569100 32552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:47.570953 32552 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:47.583670 32560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:47.584434 32558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:47.585251 32557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:47.586050 32552 server_base.cc:1034] running on GCE node
I20240501 20:08:47.586751 32552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:47.587775 32552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:47.588980 32552 hybrid_clock.cc:648] HybridClock initialized: now 1714594127588923 us; error 54 us; skew 500 ppm
I20240501 20:08:47.589345 32552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:47.591696 32552 webserver.cc:460] Webserver started at http://127.29.219.69:41931/ using document root <none> and password file <none>
I20240501 20:08:47.592458 32552 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:47.592579 32552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:47.592906 32552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:47.595630 32552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "6e30d3b5807c4d209a59e959ea666e1d"
format_stamp: "Formatted at 2024-05-01 20:08:47 on dist-test-slave-h2cj"
I20240501 20:08:47.596421 32552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "6e30d3b5807c4d209a59e959ea666e1d"
format_stamp: "Formatted at 2024-05-01 20:08:47 on dist-test-slave-h2cj"
I20240501 20:08:47.602043 32552 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 20:08:47.605527 32566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:47.606957 32552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:08:47.607122 32552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6e30d3b5807c4d209a59e959ea666e1d"
format_stamp: "Formatted at 2024-05-01 20:08:47 on dist-test-slave-h2cj"
I20240501 20:08:47.607326 32552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:47.635696 32552 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:47.636597 32552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:47.636891 32552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:47.637916 32552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:47.639515 32552 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:47.639616 32552 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:47.639735 32552 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:47.639815 32552 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:47.671892 32552 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.69:44747
I20240501 20:08:47.671959 32678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.69:44747 every 8 connection(s)
I20240501 20:08:47.673205 32552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:08:47.680502 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 32552
I20240501 20:08:47.680733 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 20:08:47.686131 32679 heartbeater.cc:347] Connected to a master server at 127.29.219.126:39585
I20240501 20:08:47.686401 32679 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:47.687213 32679 heartbeater.cc:511] Master 127.29.219.126:39585 requested a full tablet report, sending...
I20240501 20:08:47.688553 31972 ts_manager.cc:194] Registered new tserver with Master: 6e30d3b5807c4d209a59e959ea666e1d (127.29.219.69:44747)
I20240501 20:08:47.689348 31972 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.69:51537
I20240501 20:08:47.696801 30573 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 20:08:47.713375 31972 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42462:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 20:08:47.754241 32419 heartbeater.cc:503] Master 127.29.219.126:39585 was elected leader, sending a full tablet report...
I20240501 20:08:47.761371 32484 tablet_service.cc:1467] Processing CreateTablet for tablet 94afde61032b4c0dbb2e44d0034127b0 (DEFAULT_TABLE table=TestTable [id=f25717f4565f47cc8d6ffa702b4f19ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:47.761371 32093 tablet_service.cc:1467] Processing CreateTablet for tablet 94afde61032b4c0dbb2e44d0034127b0 (DEFAULT_TABLE table=TestTable [id=f25717f4565f47cc8d6ffa702b4f19ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:47.761946 32614 tablet_service.cc:1467] Processing CreateTablet for tablet 94afde61032b4c0dbb2e44d0034127b0 (DEFAULT_TABLE table=TestTable [id=f25717f4565f47cc8d6ffa702b4f19ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:47.762970 32484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94afde61032b4c0dbb2e44d0034127b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:47.762960 32093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94afde61032b4c0dbb2e44d0034127b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:47.763206 32614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94afde61032b4c0dbb2e44d0034127b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:47.773967 32697 tablet_bootstrap.cc:492] T 94afde61032b4c0dbb2e44d0034127b0 P 6e30d3b5807c4d209a59e959ea666e1d: Bootstrap starting.
I20240501 20:08:47.776436 32698 tablet_bootstrap.cc:492] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258: Bootstrap starting.
I20240501 20:08:47.778101 32697 tablet_bootstrap.cc:654] T 94afde61032b4c0dbb2e44d0034127b0 P 6e30d3b5807c4d209a59e959ea666e1d: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:47.779692 32697 log.cc:824] T 94afde61032b4c0dbb2e44d0034127b0 P 6e30d3b5807c4d209a59e959ea666e1d: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:47.779968 32698 tablet_bootstrap.cc:654] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:47.781239 32698 log.cc:824] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:47.783145 32699 tablet_bootstrap.cc:492] T 94afde61032b4c0dbb2e44d0034127b0 P 5a10add746b347f3a8a654539df2bb79: Bootstrap starting.
I20240501 20:08:47.784606 32698 tablet_bootstrap.cc:492] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258: No bootstrap required, opened a new log
I20240501 20:08:47.784916 32698 ts_tablet_manager.cc:1367] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240501 20:08:47.786738 32699 tablet_bootstrap.cc:654] T 94afde61032b4c0dbb2e44d0034127b0 P 5a10add746b347f3a8a654539df2bb79: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:47.788686 32697 tablet_bootstrap.cc:492] T 94afde61032b4c0dbb2e44d0034127b0 P 6e30d3b5807c4d209a59e959ea666e1d: No bootstrap required, opened a new log
I20240501 20:08:47.788985 32697 ts_tablet_manager.cc:1367] T 94afde61032b4c0dbb2e44d0034127b0 P 6e30d3b5807c4d209a59e959ea666e1d: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20240501 20:08:47.789552 32699 log.cc:824] T 94afde61032b4c0dbb2e44d0034127b0 P 5a10add746b347f3a8a654539df2bb79: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:47.791939 32699 tablet_bootstrap.cc:492] T 94afde61032b4c0dbb2e44d0034127b0 P 5a10add746b347f3a8a654539df2bb79: No bootstrap required, opened a new log
I20240501 20:08:47.792174 32699 ts_tablet_manager.cc:1367] T 94afde61032b4c0dbb2e44d0034127b0 P 5a10add746b347f3a8a654539df2bb79: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20240501 20:08:47.793545 32698 raft_consensus.cc:348] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7362bc6774b844a7b81931533fca8258" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37329 } } peers { permanent_uuid: "6e30d3b5807c4d209a59e959ea666e1d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44747 } } peers { permanent_uuid: "5a10add746b347f3a8a654539df2bb79" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 35361 } }
I20240501 20:08:47.793989 32698 raft_consensus.cc:374] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:47.794098 32698 raft_consensus.cc:729] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7362bc6774b844a7b81931533fca8258, State: Initialized, Role: FOLLOWER
I20240501 20:08:47.794848 32698 consensus_queue.cc:260] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7362bc6774b844a7b81931533fca8258" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37329 } } peers { permanent_uuid: "6e30d3b5807c4d209a59e959ea666e1d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44747 } } peers { permanent_uuid: "5a10add746b347f3a8a654539df2bb79" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 35361 } }
I20240501 20:08:47.796000 32549 heartbeater.cc:503] Master 127.29.219.126:39585 was elected leader, sending a full tablet report...
I20240501 20:08:47.796589 32698 ts_tablet_manager.cc:1398] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20240501 20:08:47.797408 32697 raft_consensus.cc:348] T 94afde61032b4c0dbb2e44d0034127b0 P 6e30d3b5807c4d209a59e959ea666e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7362bc6774b844a7b81931533fca8258" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37329 } } peers { permanent_uuid: "6e30d3b5807c4d209a59e959ea666e1d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44747 } } peers { permanent_uuid: "5a10add746b347f3a8a654539df2bb79" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 35361 } }
I20240501 20:08:47.797785 32697 raft_consensus.cc:374] T 94afde61032b4c0dbb2e44d0034127b0 P 6e30d3b5807c4d209a59e959ea666e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:47.797901 32697 raft_consensus.cc:729] T 94afde61032b4c0dbb2e44d0034127b0 P 6e30d3b5807c4d209a59e959ea666e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e30d3b5807c4d209a59e959ea666e1d, State: Initialized, Role: FOLLOWER
I20240501 20:08:47.798511 32697 consensus_queue.cc:260] T 94afde61032b4c0dbb2e44d0034127b0 P 6e30d3b5807c4d209a59e959ea666e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7362bc6774b844a7b81931533fca8258" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37329 } } peers { permanent_uuid: "6e30d3b5807c4d209a59e959ea666e1d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44747 } } peers { permanent_uuid: "5a10add746b347f3a8a654539df2bb79" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 35361 } }
I20240501 20:08:47.799628 32679 heartbeater.cc:503] Master 127.29.219.126:39585 was elected leader, sending a full tablet report...
I20240501 20:08:47.799731 32699 raft_consensus.cc:348] T 94afde61032b4c0dbb2e44d0034127b0 P 5a10add746b347f3a8a654539df2bb79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7362bc6774b844a7b81931533fca8258" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37329 } } peers { permanent_uuid: "6e30d3b5807c4d209a59e959ea666e1d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44747 } } peers { permanent_uuid: "5a10add746b347f3a8a654539df2bb79" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 35361 } }
I20240501 20:08:47.800170 32699 raft_consensus.cc:374] T 94afde61032b4c0dbb2e44d0034127b0 P 5a10add746b347f3a8a654539df2bb79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:47.800266 32699 raft_consensus.cc:729] T 94afde61032b4c0dbb2e44d0034127b0 P 5a10add746b347f3a8a654539df2bb79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a10add746b347f3a8a654539df2bb79, State: Initialized, Role: FOLLOWER
I20240501 20:08:47.800315 32697 ts_tablet_manager.cc:1398] T 94afde61032b4c0dbb2e44d0034127b0 P 6e30d3b5807c4d209a59e959ea666e1d: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20240501 20:08:47.800917 32699 consensus_queue.cc:260] T 94afde61032b4c0dbb2e44d0034127b0 P 5a10add746b347f3a8a654539df2bb79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7362bc6774b844a7b81931533fca8258" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37329 } } peers { permanent_uuid: "6e30d3b5807c4d209a59e959ea666e1d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44747 } } peers { permanent_uuid: "5a10add746b347f3a8a654539df2bb79" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 35361 } }
I20240501 20:08:47.802841 32699 ts_tablet_manager.cc:1398] T 94afde61032b4c0dbb2e44d0034127b0 P 5a10add746b347f3a8a654539df2bb79: Time spent starting tablet: real 0.011s user 0.004s sys 0.008s
W20240501 20:08:47.816231 32159 tablet.cc:2369] T 94afde61032b4c0dbb2e44d0034127b0 P 5a10add746b347f3a8a654539df2bb79: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:08:47.924983 32680 tablet.cc:2369] T 94afde61032b4c0dbb2e44d0034127b0 P 6e30d3b5807c4d209a59e959ea666e1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:47.928316 32703 raft_consensus.cc:482] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:08:47.928562 32703 raft_consensus.cc:504] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7362bc6774b844a7b81931533fca8258" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37329 } } peers { permanent_uuid: "6e30d3b5807c4d209a59e959ea666e1d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44747 } } peers { permanent_uuid: "5a10add746b347f3a8a654539df2bb79" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 35361 } }
I20240501 20:08:47.930238 32703 leader_election.cc:288] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e30d3b5807c4d209a59e959ea666e1d (127.29.219.69:44747), 5a10add746b347f3a8a654539df2bb79 (127.29.219.65:35361)
I20240501 20:08:47.937768 32113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94afde61032b4c0dbb2e44d0034127b0" candidate_uuid: "7362bc6774b844a7b81931533fca8258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a10add746b347f3a8a654539df2bb79" is_pre_election: true
I20240501 20:08:47.937768 32634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94afde61032b4c0dbb2e44d0034127b0" candidate_uuid: "7362bc6774b844a7b81931533fca8258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e30d3b5807c4d209a59e959ea666e1d" is_pre_election: true
I20240501 20:08:47.938382 32113 raft_consensus.cc:2454] T 94afde61032b4c0dbb2e44d0034127b0 P 5a10add746b347f3a8a654539df2bb79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7362bc6774b844a7b81931533fca8258 in term 0.
I20240501 20:08:47.938382 32634 raft_consensus.cc:2454] T 94afde61032b4c0dbb2e44d0034127b0 P 6e30d3b5807c4d209a59e959ea666e1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7362bc6774b844a7b81931533fca8258 in term 0.
I20240501 20:08:47.939097 32440 leader_election.cc:302] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e30d3b5807c4d209a59e959ea666e1d, 7362bc6774b844a7b81931533fca8258; no voters:
I20240501 20:08:47.939508 32703 raft_consensus.cc:2781] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:08:47.939644 32703 raft_consensus.cc:482] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:08:47.939788 32703 raft_consensus.cc:3037] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:47.941895 32703 raft_consensus.cc:504] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7362bc6774b844a7b81931533fca8258" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37329 } } peers { permanent_uuid: "6e30d3b5807c4d209a59e959ea666e1d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44747 } } peers { permanent_uuid: "5a10add746b347f3a8a654539df2bb79" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 35361 } }
I20240501 20:08:47.942574 32703 leader_election.cc:288] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258 [CANDIDATE]: Term 1 election: Requested vote from peers 6e30d3b5807c4d209a59e959ea666e1d (127.29.219.69:44747), 5a10add746b347f3a8a654539df2bb79 (127.29.219.65:35361)
I20240501 20:08:47.943063 32634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94afde61032b4c0dbb2e44d0034127b0" candidate_uuid: "7362bc6774b844a7b81931533fca8258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e30d3b5807c4d209a59e959ea666e1d"
I20240501 20:08:47.943202 32113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94afde61032b4c0dbb2e44d0034127b0" candidate_uuid: "7362bc6774b844a7b81931533fca8258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a10add746b347f3a8a654539df2bb79"
I20240501 20:08:47.943249 32634 raft_consensus.cc:3037] T 94afde61032b4c0dbb2e44d0034127b0 P 6e30d3b5807c4d209a59e959ea666e1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:47.943408 32113 raft_consensus.cc:3037] T 94afde61032b4c0dbb2e44d0034127b0 P 5a10add746b347f3a8a654539df2bb79 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:47.945348 32634 raft_consensus.cc:2454] T 94afde61032b4c0dbb2e44d0034127b0 P 6e30d3b5807c4d209a59e959ea666e1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7362bc6774b844a7b81931533fca8258 in term 1.
I20240501 20:08:47.945457 32113 raft_consensus.cc:2454] T 94afde61032b4c0dbb2e44d0034127b0 P 5a10add746b347f3a8a654539df2bb79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7362bc6774b844a7b81931533fca8258 in term 1.
I20240501 20:08:47.945768 32440 leader_election.cc:302] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e30d3b5807c4d209a59e959ea666e1d, 7362bc6774b844a7b81931533fca8258; no voters:
I20240501 20:08:47.946061 32703 raft_consensus.cc:2781] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:47.946468 32703 raft_consensus.cc:686] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258 [term 1 LEADER]: Becoming Leader. State: Replica: 7362bc6774b844a7b81931533fca8258, State: Running, Role: LEADER
I20240501 20:08:47.946972 32703 consensus_queue.cc:237] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7362bc6774b844a7b81931533fca8258" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37329 } } peers { permanent_uuid: "6e30d3b5807c4d209a59e959ea666e1d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44747 } } peers { permanent_uuid: "5a10add746b347f3a8a654539df2bb79" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 35361 } }
W20240501 20:08:47.950212 32550 tablet.cc:2369] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:47.952750 31971 catalog_manager.cc:5484] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7362bc6774b844a7b81931533fca8258 (127.29.219.68). New cstate: current_term: 1 leader_uuid: "7362bc6774b844a7b81931533fca8258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7362bc6774b844a7b81931533fca8258" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e30d3b5807c4d209a59e959ea666e1d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a10add746b347f3a8a654539df2bb79" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 35361 } health_report { overall_health: UNKNOWN } } }
I20240501 20:08:47.995579 30573 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 20:08:47.997689 30573 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5a10add746b347f3a8a654539df2bb79 to finish bootstrapping
I20240501 20:08:48.006124 30573 ts_itest-base.cc:233] Waiting for 0 tablets on tserver afbbb4b93f0e4ae2a43473307178a5bb to finish bootstrapping
I20240501 20:08:48.015017 30573 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a1cf2ef277f34512925b234044ea6521 to finish bootstrapping
I20240501 20:08:48.023938 30573 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7362bc6774b844a7b81931533fca8258 to finish bootstrapping
I20240501 20:08:48.031854 30573 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6e30d3b5807c4d209a59e959ea666e1d to finish bootstrapping
I20240501 20:08:48.039074 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 32031
I20240501 20:08:48.040079 32153 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:48.127127 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 32031
W20240501 20:08:48.403170 32439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.219.65:35361: connect: Connection refused (error 111)
W20240501 20:08:48.404527 32439 consensus_peers.cc:485] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258 -> Peer 5a10add746b347f3a8a654539df2bb79 (127.29.219.65:35361): Couldn't send request to peer 5a10add746b347f3a8a654539df2bb79. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.65:35361: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:08:48.523010 32703 consensus_queue.cc:1035] T 94afde61032b4c0dbb2e44d0034127b0 P 7362bc6774b844a7b81931533fca8258 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e30d3b5807c4d209a59e959ea666e1d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:08:48.626441 32484 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:48.627296 32614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:48.635263 32354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:48.635762 32224 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:48.792660 32614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:48.796975 32484 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:48.799757 32224 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:48.808738 32354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:48.957521 32614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:48.958112 32484 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:48.981535 32354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:48.985808 32224 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:49.630833 32614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:49.635069 32354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:49.635187 32484 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:49.644837 32224 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:49.771376 32614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:49.777078 32484 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:49.781162 32354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:49.783169 32224 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:49.947399 32614 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:49.955046 32354 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:49.956077 32484 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:08:49.956102 32224 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:08:50.106339 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 32162
I20240501 20:08:50.107704 32284 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:50.205039 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 32162
I20240501 20:08:50.226343 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 32292
I20240501 20:08:50.227456 32413 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:50.320971 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 32292
I20240501 20:08:50.339481 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 32422
I20240501 20:08:50.340662 32544 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:50.438848 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 32422
I20240501 20:08:50.457582 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 32552
I20240501 20:08:50.458788 32674 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:50.553993 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 32552
W20240501 20:08:50.574290 31959 connection.cc:535] server connection from 127.29.219.69:51537 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:08:50.574425 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 31941
I20240501 20:08:50.575534 32002 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:50.692265 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 31941
2024-05-01T20:08:50Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/1 (5966 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5966 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/1
I20240501 20:08:50.736768 30573 test_util.cc:268] Using random seed: 236500581
I20240501 20:08:50.738785 30573 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:08:50.738864 30573 ts_itest-base.cc:113] --------------
I20240501 20:08:50.738929 30573 ts_itest-base.cc:114] 7 tablet servers
I20240501 20:08:50.738983 30573 ts_itest-base.cc:115] 1 replicas per TS
I20240501 20:08:50.739050 30573 ts_itest-base.cc:116] --------------
2024-05-01T20:08:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:08:50Z Disabled control of system clock
I20240501 20:08:50.765280 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.219.126:46839
--webserver_interface=127.29.219.126
--webserver_port=0
--builtin_ntp_servers=127.29.219.84:42611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.219.126:46839
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 20:08:51.100514 453 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 20:08:51.100883 453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:51.100958 453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:51.101033 453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:51.108673 453 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:51.108819 453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:08:51.108877 453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:51.108923 453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:08:51.108989 453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:08:51.118348 453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:42611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.29.219.126:46839
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.219.126:46839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.29.219.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:51.119798 453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:51.121781 453 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:51.133771 459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:51.133800 458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:51.134510 461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:51.135643 453 server_base.cc:1034] running on GCE node
I20240501 20:08:51.136502 453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:51.138087 453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:51.139304 453 hybrid_clock.cc:648] HybridClock initialized: now 1714594131139226 us; error 58 us; skew 500 ppm
I20240501 20:08:51.139629 453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:51.148265 453 webserver.cc:460] Webserver started at http://127.29.219.126:44689/ using document root <none> and password file <none>
I20240501 20:08:51.149073 453 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:51.149195 453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:51.149469 453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:51.152319 453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ef92b2d391194ecf9cfbe91fe77708c7"
format_stamp: "Formatted at 2024-05-01 20:08:51 on dist-test-slave-h2cj"
I20240501 20:08:51.153098 453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ef92b2d391194ecf9cfbe91fe77708c7"
format_stamp: "Formatted at 2024-05-01 20:08:51 on dist-test-slave-h2cj"
I20240501 20:08:51.158689 453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 20:08:51.162271 467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:51.163780 453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240501 20:08:51.163976 453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ef92b2d391194ecf9cfbe91fe77708c7"
format_stamp: "Formatted at 2024-05-01 20:08:51 on dist-test-slave-h2cj"
I20240501 20:08:51.164181 453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:51.180850 453 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:51.181746 453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:51.182049 453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:51.199643 453 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.126:46839
I20240501 20:08:51.199666 518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.126:46839 every 8 connection(s)
I20240501 20:08:51.201153 453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:08:51.204685 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 453
I20240501 20:08:51.204883 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:08:51.205075 519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:51.218515 519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef92b2d391194ecf9cfbe91fe77708c7: Bootstrap starting.
I20240501 20:08:51.221822 519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef92b2d391194ecf9cfbe91fe77708c7: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:51.222884 519 log.cc:824] T 00000000000000000000000000000000 P ef92b2d391194ecf9cfbe91fe77708c7: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:51.225376 519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef92b2d391194ecf9cfbe91fe77708c7: No bootstrap required, opened a new log
I20240501 20:08:51.231894 519 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef92b2d391194ecf9cfbe91fe77708c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef92b2d391194ecf9cfbe91fe77708c7" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 46839 } }
I20240501 20:08:51.232246 519 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef92b2d391194ecf9cfbe91fe77708c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:51.232327 519 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef92b2d391194ecf9cfbe91fe77708c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef92b2d391194ecf9cfbe91fe77708c7, State: Initialized, Role: FOLLOWER
I20240501 20:08:51.232926 519 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef92b2d391194ecf9cfbe91fe77708c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef92b2d391194ecf9cfbe91fe77708c7" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 46839 } }
I20240501 20:08:51.233119 519 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef92b2d391194ecf9cfbe91fe77708c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:51.233214 519 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef92b2d391194ecf9cfbe91fe77708c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:51.233336 519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef92b2d391194ecf9cfbe91fe77708c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:51.235455 519 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef92b2d391194ecf9cfbe91fe77708c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef92b2d391194ecf9cfbe91fe77708c7" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 46839 } }
I20240501 20:08:51.235991 519 leader_election.cc:302] T 00000000000000000000000000000000 P ef92b2d391194ecf9cfbe91fe77708c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef92b2d391194ecf9cfbe91fe77708c7; no voters:
I20240501 20:08:51.236742 519 leader_election.cc:288] T 00000000000000000000000000000000 P ef92b2d391194ecf9cfbe91fe77708c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:51.236876 524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef92b2d391194ecf9cfbe91fe77708c7 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:51.237612 524 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef92b2d391194ecf9cfbe91fe77708c7 [term 1 LEADER]: Becoming Leader. State: Replica: ef92b2d391194ecf9cfbe91fe77708c7, State: Running, Role: LEADER
I20240501 20:08:51.238258 524 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef92b2d391194ecf9cfbe91fe77708c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef92b2d391194ecf9cfbe91fe77708c7" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 46839 } }
I20240501 20:08:51.239322 519 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef92b2d391194ecf9cfbe91fe77708c7 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:08:51.241600 526 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef92b2d391194ecf9cfbe91fe77708c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef92b2d391194ecf9cfbe91fe77708c7. Latest consensus state: current_term: 1 leader_uuid: "ef92b2d391194ecf9cfbe91fe77708c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef92b2d391194ecf9cfbe91fe77708c7" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 46839 } } }
I20240501 20:08:51.241906 526 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef92b2d391194ecf9cfbe91fe77708c7 [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:51.241744 525 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef92b2d391194ecf9cfbe91fe77708c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef92b2d391194ecf9cfbe91fe77708c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef92b2d391194ecf9cfbe91fe77708c7" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 46839 } } }
I20240501 20:08:51.242980 525 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef92b2d391194ecf9cfbe91fe77708c7 [sys.catalog]: This master's current role is: LEADER
I20240501 20:08:51.244449 530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:08:51.250581 530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:08:51.260398 530 catalog_manager.cc:1347] Generated new cluster ID: 46ef35479de1427a8fb7530880b91be2
I20240501 20:08:51.260532 530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:08:51.283548 530 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:08:51.284807 530 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:08:51.304059 530 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ef92b2d391194ecf9cfbe91fe77708c7: Generated new TSK 0
I20240501 20:08:51.305262 530 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:08:51.319070 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.65:0
--local_ip_for_outbound_sockets=127.29.219.65
--webserver_interface=127.29.219.65
--webserver_port=0
--tserver_master_addrs=127.29.219.126:46839
--builtin_ntp_servers=127.29.219.84:42611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:08:51.660159 543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:51.660488 543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:51.660632 543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:51.668349 543 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:51.668504 543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:51.668664 543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.65
I20240501 20:08:51.678136 543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:42611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.29.219.65
--webserver_port=0
--tserver_master_addrs=127.29.219.126:46839
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.65
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:51.679606 543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:51.681353 543 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:51.693379 549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:51.694379 548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:51.694499 551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:51.696271 543 server_base.cc:1034] running on GCE node
I20240501 20:08:51.696899 543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:51.697865 543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:51.699080 543 hybrid_clock.cc:648] HybridClock initialized: now 1714594131699024 us; error 45 us; skew 500 ppm
I20240501 20:08:51.699399 543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:51.701771 543 webserver.cc:460] Webserver started at http://127.29.219.65:33791/ using document root <none> and password file <none>
I20240501 20:08:51.702589 543 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:51.702749 543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:51.703064 543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:51.705694 543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "c0931967d12847e08d9ff883c85f4237"
format_stamp: "Formatted at 2024-05-01 20:08:51 on dist-test-slave-h2cj"
I20240501 20:08:51.706467 543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "c0931967d12847e08d9ff883c85f4237"
format_stamp: "Formatted at 2024-05-01 20:08:51 on dist-test-slave-h2cj"
I20240501 20:08:51.711970 543 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240501 20:08:51.715588 557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:51.717056 543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:08:51.717212 543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "c0931967d12847e08d9ff883c85f4237"
format_stamp: "Formatted at 2024-05-01 20:08:51 on dist-test-slave-h2cj"
I20240501 20:08:51.717427 543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:51.733654 543 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:51.734531 543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:51.734846 543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:51.735801 543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:51.737425 543 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:51.737529 543 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:51.737651 543 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:51.737728 543 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:51.771003 543 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.65:36897
I20240501 20:08:51.771062 670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.65:36897 every 8 connection(s)
I20240501 20:08:51.772372 543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:08:51.779258 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 543
I20240501 20:08:51.779487 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:08:51.783083 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.66:0
--local_ip_for_outbound_sockets=127.29.219.66
--webserver_interface=127.29.219.66
--webserver_port=0
--tserver_master_addrs=127.29.219.126:46839
--builtin_ntp_servers=127.29.219.84:42611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:51.786556 671 heartbeater.cc:347] Connected to a master server at 127.29.219.126:46839
I20240501 20:08:51.786942 671 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:51.787825 671 heartbeater.cc:511] Master 127.29.219.126:46839 requested a full tablet report, sending...
I20240501 20:08:51.789942 484 ts_manager.cc:194] Registered new tserver with Master: c0931967d12847e08d9ff883c85f4237 (127.29.219.65:36897)
I20240501 20:08:51.791561 484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.65:41217
W20240501 20:08:52.128017 675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:52.128326 675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:52.128443 675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:52.136176 675 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:52.136320 675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:52.136474 675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.66
I20240501 20:08:52.145861 675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:42611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.29.219.66
--webserver_port=0
--tserver_master_addrs=127.29.219.126:46839
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.66
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:52.147274 675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:52.149206 675 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:52.161991 680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:52.162863 683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:52.163580 681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:52.164806 675 server_base.cc:1034] running on GCE node
I20240501 20:08:52.165509 675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:52.166577 675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:52.167904 675 hybrid_clock.cc:648] HybridClock initialized: now 1714594132167839 us; error 51 us; skew 500 ppm
I20240501 20:08:52.168413 675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:52.171058 675 webserver.cc:460] Webserver started at http://127.29.219.66:34719/ using document root <none> and password file <none>
I20240501 20:08:52.171893 675 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:52.172019 675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:52.172358 675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:52.175251 675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b811afcdb9d34a38b7e14e693b58093e"
format_stamp: "Formatted at 2024-05-01 20:08:52 on dist-test-slave-h2cj"
I20240501 20:08:52.176069 675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b811afcdb9d34a38b7e14e693b58093e"
format_stamp: "Formatted at 2024-05-01 20:08:52 on dist-test-slave-h2cj"
I20240501 20:08:52.181733 675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 20:08:52.185735 689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:52.187472 675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 20:08:52.187688 675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b811afcdb9d34a38b7e14e693b58093e"
format_stamp: "Formatted at 2024-05-01 20:08:52 on dist-test-slave-h2cj"
I20240501 20:08:52.187927 675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:52.214529 675 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:52.215708 675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:52.216029 675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:52.217247 675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:52.219267 675 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:52.219379 675 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:52.219496 675 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:52.219575 675 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:52.254892 675 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.66:40067
I20240501 20:08:52.254976 802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.66:40067 every 8 connection(s)
I20240501 20:08:52.256312 675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:08:52.264621 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 675
I20240501 20:08:52.264787 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:08:52.268590 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.67:0
--local_ip_for_outbound_sockets=127.29.219.67
--webserver_interface=127.29.219.67
--webserver_port=0
--tserver_master_addrs=127.29.219.126:46839
--builtin_ntp_servers=127.29.219.84:42611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:52.270198 803 heartbeater.cc:347] Connected to a master server at 127.29.219.126:46839
I20240501 20:08:52.270665 803 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:52.271911 803 heartbeater.cc:511] Master 127.29.219.126:46839 requested a full tablet report, sending...
I20240501 20:08:52.273682 484 ts_manager.cc:194] Registered new tserver with Master: b811afcdb9d34a38b7e14e693b58093e (127.29.219.66:40067)
I20240501 20:08:52.274910 484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.66:53585
W20240501 20:08:52.617966 806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:52.618291 806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:52.618387 806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:52.626345 806 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:52.626528 806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:52.626758 806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.67
I20240501 20:08:52.636277 806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:42611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.29.219.67
--webserver_port=0
--tserver_master_addrs=127.29.219.126:46839
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.67
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:52.637789 806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:52.639825 806 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:52.653887 814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:52.654912 812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:52.654987 811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:52.655830 806 server_base.cc:1034] running on GCE node
I20240501 20:08:52.656517 806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:52.657584 806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:52.658843 806 hybrid_clock.cc:648] HybridClock initialized: now 1714594132658788 us; error 79 us; skew 500 ppm
I20240501 20:08:52.659272 806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:52.662252 806 webserver.cc:460] Webserver started at http://127.29.219.67:36873/ using document root <none> and password file <none>
I20240501 20:08:52.663118 806 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:52.663228 806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:52.663563 806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:52.666574 806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "be64c56dce3d46fa80ba75efff719a7d"
format_stamp: "Formatted at 2024-05-01 20:08:52 on dist-test-slave-h2cj"
I20240501 20:08:52.667527 806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "be64c56dce3d46fa80ba75efff719a7d"
format_stamp: "Formatted at 2024-05-01 20:08:52 on dist-test-slave-h2cj"
I20240501 20:08:52.673398 806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:08:52.677313 820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:52.679092 806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:08:52.679282 806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "be64c56dce3d46fa80ba75efff719a7d"
format_stamp: "Formatted at 2024-05-01 20:08:52 on dist-test-slave-h2cj"
I20240501 20:08:52.679497 806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:52.711386 806 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:52.712373 806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:52.712678 806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:52.713771 806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:52.715741 806 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:52.715835 806 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:52.715950 806 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:52.716023 806 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:52.751472 806 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.67:42219
I20240501 20:08:52.751536 933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.67:42219 every 8 connection(s)
I20240501 20:08:52.752872 806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:08:52.761138 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 806
I20240501 20:08:52.761322 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:08:52.765028 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.68:0
--local_ip_for_outbound_sockets=127.29.219.68
--webserver_interface=127.29.219.68
--webserver_port=0
--tserver_master_addrs=127.29.219.126:46839
--builtin_ntp_servers=127.29.219.84:42611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:52.767253 934 heartbeater.cc:347] Connected to a master server at 127.29.219.126:46839
I20240501 20:08:52.767612 934 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:52.768610 934 heartbeater.cc:511] Master 127.29.219.126:46839 requested a full tablet report, sending...
I20240501 20:08:52.770634 484 ts_manager.cc:194] Registered new tserver with Master: be64c56dce3d46fa80ba75efff719a7d (127.29.219.67:42219)
I20240501 20:08:52.771883 484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.67:55745
I20240501 20:08:52.795154 671 heartbeater.cc:503] Master 127.29.219.126:46839 was elected leader, sending a full tablet report...
W20240501 20:08:53.121376 937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:53.121757 937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:53.121924 937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:53.129750 937 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:53.129910 937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:53.130089 937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.68
I20240501 20:08:53.139868 937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:42611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.29.219.68
--webserver_port=0
--tserver_master_addrs=127.29.219.126:46839
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.68
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:53.141345 937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:53.143221 937 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:53.156735 942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:53.157752 943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:53.159001 945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:53.159246 937 server_base.cc:1034] running on GCE node
I20240501 20:08:53.160068 937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:53.161167 937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:53.162389 937 hybrid_clock.cc:648] HybridClock initialized: now 1714594133162338 us; error 48 us; skew 500 ppm
I20240501 20:08:53.162796 937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:53.165182 937 webserver.cc:460] Webserver started at http://127.29.219.68:34497/ using document root <none> and password file <none>
I20240501 20:08:53.166028 937 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:53.166146 937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:53.166464 937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:53.169318 937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "52541d349ee24edaac72825773fc848a"
format_stamp: "Formatted at 2024-05-01 20:08:53 on dist-test-slave-h2cj"
I20240501 20:08:53.170123 937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "52541d349ee24edaac72825773fc848a"
format_stamp: "Formatted at 2024-05-01 20:08:53 on dist-test-slave-h2cj"
I20240501 20:08:53.175792 937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240501 20:08:53.179631 951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:53.181250 937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:08:53.181439 937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "52541d349ee24edaac72825773fc848a"
format_stamp: "Formatted at 2024-05-01 20:08:53 on dist-test-slave-h2cj"
I20240501 20:08:53.181676 937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:53.199048 937 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:53.200183 937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:53.200495 937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:53.201756 937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:53.203828 937 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:53.203940 937 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:53.204047 937 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:53.204115 937 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:53.241816 937 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.68:46431
I20240501 20:08:53.241883 1064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.68:46431 every 8 connection(s)
I20240501 20:08:53.243185 937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:08:53.247627 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 937
I20240501 20:08:53.247871 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:08:53.251361 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.69:0
--local_ip_for_outbound_sockets=127.29.219.69
--webserver_interface=127.29.219.69
--webserver_port=0
--tserver_master_addrs=127.29.219.126:46839
--builtin_ntp_servers=127.29.219.84:42611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:53.258610 1065 heartbeater.cc:347] Connected to a master server at 127.29.219.126:46839
I20240501 20:08:53.259074 1065 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:53.259958 1065 heartbeater.cc:511] Master 127.29.219.126:46839 requested a full tablet report, sending...
I20240501 20:08:53.261660 484 ts_manager.cc:194] Registered new tserver with Master: 52541d349ee24edaac72825773fc848a (127.29.219.68:46431)
I20240501 20:08:53.262478 484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.68:33747
I20240501 20:08:53.277551 803 heartbeater.cc:503] Master 127.29.219.126:46839 was elected leader, sending a full tablet report...
W20240501 20:08:53.598836 1068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:53.599184 1068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:53.599330 1068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:53.607002 1068 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:53.607168 1068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:53.607334 1068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.69
I20240501 20:08:53.616629 1068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:42611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.29.219.69
--webserver_port=0
--tserver_master_addrs=127.29.219.126:46839
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.69
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:53.618062 1068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:53.619869 1068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:53.632869 1073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:53.633762 1076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:53.634851 1074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:53.635722 1068 server_base.cc:1034] running on GCE node
I20240501 20:08:53.636445 1068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:53.637553 1068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:53.638784 1068 hybrid_clock.cc:648] HybridClock initialized: now 1714594133638732 us; error 49 us; skew 500 ppm
I20240501 20:08:53.639182 1068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:53.641767 1068 webserver.cc:460] Webserver started at http://127.29.219.69:38001/ using document root <none> and password file <none>
I20240501 20:08:53.642571 1068 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:53.642737 1068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:53.643074 1068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:53.646003 1068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "efa05351dbe344c7a451e7fdbeed47f0"
format_stamp: "Formatted at 2024-05-01 20:08:53 on dist-test-slave-h2cj"
I20240501 20:08:53.646893 1068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "efa05351dbe344c7a451e7fdbeed47f0"
format_stamp: "Formatted at 2024-05-01 20:08:53 on dist-test-slave-h2cj"
I20240501 20:08:53.652863 1068 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240501 20:08:53.656888 1082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:53.658501 1068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:08:53.658777 1068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "efa05351dbe344c7a451e7fdbeed47f0"
format_stamp: "Formatted at 2024-05-01 20:08:53 on dist-test-slave-h2cj"
I20240501 20:08:53.659003 1068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:53.676335 1068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:53.677263 1068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:53.677554 1068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:53.678579 1068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:53.680207 1068 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:53.680320 1068 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:53.680434 1068 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:53.680508 1068 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:53.713692 1068 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.69:44913
I20240501 20:08:53.713742 1195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.69:44913 every 8 connection(s)
I20240501 20:08:53.715166 1068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:08:53.725412 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 1068
I20240501 20:08:53.725647 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 20:08:53.729017 1196 heartbeater.cc:347] Connected to a master server at 127.29.219.126:46839
I20240501 20:08:53.729032 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.70:0
--local_ip_for_outbound_sockets=127.29.219.70
--webserver_interface=127.29.219.70
--webserver_port=0
--tserver_master_addrs=127.29.219.126:46839
--builtin_ntp_servers=127.29.219.84:42611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:53.729341 1196 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:53.730185 1196 heartbeater.cc:511] Master 127.29.219.126:46839 requested a full tablet report, sending...
I20240501 20:08:53.731642 484 ts_manager.cc:194] Registered new tserver with Master: efa05351dbe344c7a451e7fdbeed47f0 (127.29.219.69:44913)
I20240501 20:08:53.732424 484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.69:47445
I20240501 20:08:53.775924 934 heartbeater.cc:503] Master 127.29.219.126:46839 was elected leader, sending a full tablet report...
W20240501 20:08:54.087440 1200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:54.087831 1200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:54.087963 1200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:54.095613 1200 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:54.095747 1200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:54.095908 1200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.70
I20240501 20:08:54.105870 1200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:42611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.29.219.70
--webserver_port=0
--tserver_master_addrs=127.29.219.126:46839
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.70
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:54.107290 1200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:54.109129 1200 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:54.122831 1208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:54.122910 1205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:54.124082 1206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:54.125038 1200 server_base.cc:1034] running on GCE node
I20240501 20:08:54.125835 1200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:54.126941 1200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:54.128191 1200 hybrid_clock.cc:648] HybridClock initialized: now 1714594134128133 us; error 47 us; skew 500 ppm
I20240501 20:08:54.128577 1200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:54.131803 1200 webserver.cc:460] Webserver started at http://127.29.219.70:39483/ using document root <none> and password file <none>
I20240501 20:08:54.132577 1200 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:54.132706 1200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:54.133028 1200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:54.135869 1200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "09cd35a57c9d4c0f86544e6ac1a55279"
format_stamp: "Formatted at 2024-05-01 20:08:54 on dist-test-slave-h2cj"
I20240501 20:08:54.137328 1200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "09cd35a57c9d4c0f86544e6ac1a55279"
format_stamp: "Formatted at 2024-05-01 20:08:54 on dist-test-slave-h2cj"
I20240501 20:08:54.143388 1200 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240501 20:08:54.147150 1214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:54.148744 1200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 20:08:54.148923 1200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "09cd35a57c9d4c0f86544e6ac1a55279"
format_stamp: "Formatted at 2024-05-01 20:08:54 on dist-test-slave-h2cj"
I20240501 20:08:54.149132 1200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:54.171185 1200 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:54.172128 1200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:54.172430 1200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:54.173434 1200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:54.175069 1200 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:54.175176 1200 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:54.175297 1200 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:54.175370 1200 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:54.210069 1200 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.70:40955
I20240501 20:08:54.210100 1327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.70:40955 every 8 connection(s)
I20240501 20:08:54.211486 1200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:08:54.220463 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 1200
I20240501 20:08:54.220640 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 20:08:54.223999 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.71:0
--local_ip_for_outbound_sockets=127.29.219.71
--webserver_interface=127.29.219.71
--webserver_port=0
--tserver_master_addrs=127.29.219.126:46839
--builtin_ntp_servers=127.29.219.84:42611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:08:54.225070 1328 heartbeater.cc:347] Connected to a master server at 127.29.219.126:46839
I20240501 20:08:54.225437 1328 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:54.226537 1328 heartbeater.cc:511] Master 127.29.219.126:46839 requested a full tablet report, sending...
I20240501 20:08:54.228219 484 ts_manager.cc:194] Registered new tserver with Master: 09cd35a57c9d4c0f86544e6ac1a55279 (127.29.219.70:40955)
I20240501 20:08:54.229061 484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.70:47993
I20240501 20:08:54.265156 1065 heartbeater.cc:503] Master 127.29.219.126:46839 was elected leader, sending a full tablet report...
W20240501 20:08:54.573213 1331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:54.573529 1331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:08:54.573669 1331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:54.581262 1331 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:08:54.581418 1331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:08:54.581640 1331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.71
I20240501 20:08:54.591063 1331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:42611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.29.219.71
--webserver_port=0
--tserver_master_addrs=127.29.219.126:46839
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.71
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:08:54.592502 1331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:08:54.594434 1331 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:08:54.606598 1336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:54.607487 1337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:08:54.608249 1339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:08:54.608995 1331 server_base.cc:1034] running on GCE node
I20240501 20:08:54.609618 1331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:08:54.610704 1331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:08:54.611917 1331 hybrid_clock.cc:648] HybridClock initialized: now 1714594134611878 us; error 31 us; skew 500 ppm
I20240501 20:08:54.612244 1331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:08:54.614557 1331 webserver.cc:460] Webserver started at http://127.29.219.71:40055/ using document root <none> and password file <none>
I20240501 20:08:54.615386 1331 fs_manager.cc:359] Metadata directory not provided
I20240501 20:08:54.615507 1331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:08:54.615821 1331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:08:54.618520 1331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "52675e825f834cf3ac80a5777be65973"
format_stamp: "Formatted at 2024-05-01 20:08:54 on dist-test-slave-h2cj"
I20240501 20:08:54.619422 1331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "52675e825f834cf3ac80a5777be65973"
format_stamp: "Formatted at 2024-05-01 20:08:54 on dist-test-slave-h2cj"
I20240501 20:08:54.625078 1331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:08:54.628690 1345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:54.630182 1331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:08:54.630342 1331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "52675e825f834cf3ac80a5777be65973"
format_stamp: "Formatted at 2024-05-01 20:08:54 on dist-test-slave-h2cj"
I20240501 20:08:54.630542 1331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:08:54.654464 1331 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:08:54.655388 1331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:08:54.655650 1331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:08:54.656654 1331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:08:54.658186 1331 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:08:54.658286 1331 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:54.658401 1331 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:08:54.658476 1331 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:08:54.689812 1331 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.71:39829
I20240501 20:08:54.689851 1458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.71:39829 every 8 connection(s)
I20240501 20:08:54.691304 1331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 20:08:54.694952 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 1331
I20240501 20:08:54.695196 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 20:08:54.705497 1459 heartbeater.cc:347] Connected to a master server at 127.29.219.126:46839
I20240501 20:08:54.705803 1459 heartbeater.cc:464] Registering TS with master...
I20240501 20:08:54.706598 1459 heartbeater.cc:511] Master 127.29.219.126:46839 requested a full tablet report, sending...
I20240501 20:08:54.708110 484 ts_manager.cc:194] Registered new tserver with Master: 52675e825f834cf3ac80a5777be65973 (127.29.219.71:39829)
I20240501 20:08:54.708995 484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.71:35809
I20240501 20:08:54.711966 30573 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 20:08:54.729269 484 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57448:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 20:08:54.735493 1196 heartbeater.cc:503] Master 127.29.219.126:46839 was elected leader, sending a full tablet report...
I20240501 20:08:54.758993 737 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef1cdbd7aa34985adb56e32c6eb64df (DEFAULT_TABLE table=TestTable [id=0c3d80d4ff1b4fb1b2b798af7ff2022c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:54.760409 737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ef1cdbd7aa34985adb56e32c6eb64df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:54.771541 1473 tablet_bootstrap.cc:492] T 1ef1cdbd7aa34985adb56e32c6eb64df P b811afcdb9d34a38b7e14e693b58093e: Bootstrap starting.
I20240501 20:08:54.775419 1473 tablet_bootstrap.cc:654] T 1ef1cdbd7aa34985adb56e32c6eb64df P b811afcdb9d34a38b7e14e693b58093e: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:54.776472 1473 log.cc:824] T 1ef1cdbd7aa34985adb56e32c6eb64df P b811afcdb9d34a38b7e14e693b58093e: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:54.779035 1473 tablet_bootstrap.cc:492] T 1ef1cdbd7aa34985adb56e32c6eb64df P b811afcdb9d34a38b7e14e693b58093e: No bootstrap required, opened a new log
I20240501 20:08:54.779317 1473 ts_tablet_manager.cc:1367] T 1ef1cdbd7aa34985adb56e32c6eb64df P b811afcdb9d34a38b7e14e693b58093e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240501 20:08:54.785526 1473 raft_consensus.cc:348] T 1ef1cdbd7aa34985adb56e32c6eb64df P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:08:54.785898 1473 raft_consensus.cc:374] T 1ef1cdbd7aa34985adb56e32c6eb64df P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:54.786007 1473 raft_consensus.cc:729] T 1ef1cdbd7aa34985adb56e32c6eb64df P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b811afcdb9d34a38b7e14e693b58093e, State: Initialized, Role: FOLLOWER
I20240501 20:08:54.786713 1473 consensus_queue.cc:260] T 1ef1cdbd7aa34985adb56e32c6eb64df P b811afcdb9d34a38b7e14e693b58093e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:08:54.786918 1473 raft_consensus.cc:388] T 1ef1cdbd7aa34985adb56e32c6eb64df P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:54.787045 1473 raft_consensus.cc:482] T 1ef1cdbd7aa34985adb56e32c6eb64df P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:54.787202 1473 raft_consensus.cc:3037] T 1ef1cdbd7aa34985adb56e32c6eb64df P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:54.839624 1473 raft_consensus.cc:504] T 1ef1cdbd7aa34985adb56e32c6eb64df P b811afcdb9d34a38b7e14e693b58093e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:08:54.840291 1473 leader_election.cc:302] T 1ef1cdbd7aa34985adb56e32c6eb64df P b811afcdb9d34a38b7e14e693b58093e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b811afcdb9d34a38b7e14e693b58093e; no voters:
I20240501 20:08:54.841028 1473 leader_election.cc:288] T 1ef1cdbd7aa34985adb56e32c6eb64df P b811afcdb9d34a38b7e14e693b58093e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:54.841207 1475 raft_consensus.cc:2781] T 1ef1cdbd7aa34985adb56e32c6eb64df P b811afcdb9d34a38b7e14e693b58093e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:54.843020 1473 ts_tablet_manager.cc:1398] T 1ef1cdbd7aa34985adb56e32c6eb64df P b811afcdb9d34a38b7e14e693b58093e: Time spent starting tablet: real 0.064s user 0.014s sys 0.000s
I20240501 20:08:54.843158 1475 raft_consensus.cc:686] T 1ef1cdbd7aa34985adb56e32c6eb64df P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Becoming Leader. State: Replica: b811afcdb9d34a38b7e14e693b58093e, State: Running, Role: LEADER
I20240501 20:08:54.843691 1475 consensus_queue.cc:237] T 1ef1cdbd7aa34985adb56e32c6eb64df P b811afcdb9d34a38b7e14e693b58093e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:08:54.850054 484 catalog_manager.cc:5484] T 1ef1cdbd7aa34985adb56e32c6eb64df P b811afcdb9d34a38b7e14e693b58093e reported cstate change: term changed from 0 to 1, leader changed from <none> to b811afcdb9d34a38b7e14e693b58093e (127.29.219.66). New cstate: current_term: 1 leader_uuid: "b811afcdb9d34a38b7e14e693b58093e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:54.880345 30573 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 20:08:54.882726 30573 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c0931967d12847e08d9ff883c85f4237 to finish bootstrapping
I20240501 20:08:54.892686 30573 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b811afcdb9d34a38b7e14e693b58093e to finish bootstrapping
I20240501 20:08:54.901978 30573 ts_itest-base.cc:233] Waiting for 0 tablets on tserver be64c56dce3d46fa80ba75efff719a7d to finish bootstrapping
I20240501 20:08:54.911475 30573 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 52541d349ee24edaac72825773fc848a to finish bootstrapping
I20240501 20:08:54.920769 30573 ts_itest-base.cc:233] Waiting for 0 tablets on tserver efa05351dbe344c7a451e7fdbeed47f0 to finish bootstrapping
I20240501 20:08:54.930404 30573 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 09cd35a57c9d4c0f86544e6ac1a55279 to finish bootstrapping
I20240501 20:08:54.940469 30573 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 52675e825f834cf3ac80a5777be65973 to finish bootstrapping
I20240501 20:08:54.951632 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 806
I20240501 20:08:54.952705 929 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:55.035864 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 806
I20240501 20:08:55.053779 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 937
I20240501 20:08:55.055059 1060 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:55.137414 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 937
I20240501 20:08:55.154372 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 1068
I20240501 20:08:55.156394 1191 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:55.232319 1328 heartbeater.cc:503] Master 127.29.219.126:46839 was elected leader, sending a full tablet report...
I20240501 20:08:55.249421 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 1068
I20240501 20:08:55.267063 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 1200
I20240501 20:08:55.268276 1323 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:55.352412 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 1200
I20240501 20:08:55.371556 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 1331
I20240501 20:08:55.372630 1454 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:08:55.455433 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 1331
I20240501 20:08:59.223776 484 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57448:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:08:59.242425 604 tablet_service.cc:1467] Processing CreateTablet for tablet 3a1f02449b91442a897ac2278ee98e29 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c30633c72e73448b81efc5fc64800aa6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:59.242492 603 tablet_service.cc:1467] Processing CreateTablet for tablet 76cd9ee2deb14a5aa9bb280b0a8dec53 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c30633c72e73448b81efc5fc64800aa6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:59.242514 605 tablet_service.cc:1467] Processing CreateTablet for tablet 2ad6ed0144eb4b619964ee4f86b9a842 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c30633c72e73448b81efc5fc64800aa6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:59.244325 604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a1f02449b91442a897ac2278ee98e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:59.244836 603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76cd9ee2deb14a5aa9bb280b0a8dec53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:59.245116 605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ad6ed0144eb4b619964ee4f86b9a842. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:59.257547 1498 tablet_bootstrap.cc:492] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237: Bootstrap starting.
I20240501 20:08:59.261081 1498 tablet_bootstrap.cc:654] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:59.262295 1498 log.cc:824] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237: Log is configured to *not* fsync() on all Append() calls
I20240501 20:08:59.265061 1498 tablet_bootstrap.cc:492] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237: No bootstrap required, opened a new log
I20240501 20:08:59.265344 1498 ts_tablet_manager.cc:1367] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240501 20:08:59.271996 1498 raft_consensus.cc:348] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:08:59.272362 1498 raft_consensus.cc:374] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:59.272437 1498 raft_consensus.cc:729] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0931967d12847e08d9ff883c85f4237, State: Initialized, Role: FOLLOWER
I20240501 20:08:59.273064 1498 consensus_queue.cc:260] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:08:59.273267 1498 raft_consensus.cc:388] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:59.273372 1498 raft_consensus.cc:482] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:59.273516 1498 raft_consensus.cc:3037] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:59.275818 1498 raft_consensus.cc:504] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:08:59.276329 1498 leader_election.cc:302] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0931967d12847e08d9ff883c85f4237; no voters:
I20240501 20:08:59.276937 1498 leader_election.cc:288] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:59.277051 1500 raft_consensus.cc:2781] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:59.277722 1500 raft_consensus.cc:686] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Becoming Leader. State: Replica: c0931967d12847e08d9ff883c85f4237, State: Running, Role: LEADER
I20240501 20:08:59.278244 1500 consensus_queue.cc:237] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:08:59.278805 1498 ts_tablet_manager.cc:1398] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20240501 20:08:59.279158 1498 tablet_bootstrap.cc:492] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237: Bootstrap starting.
W20240501 20:08:59.279664 672 tablet.cc:2369] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:08:59.281040 1498 tablet_bootstrap.cc:654] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:59.283270 1498 tablet_bootstrap.cc:492] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237: No bootstrap required, opened a new log
I20240501 20:08:59.283429 1498 ts_tablet_manager.cc:1367] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20240501 20:08:59.284044 484 catalog_manager.cc:5484] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0931967d12847e08d9ff883c85f4237 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "c0931967d12847e08d9ff883c85f4237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:59.284207 1498 raft_consensus.cc:348] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:08:59.284375 1498 raft_consensus.cc:374] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:59.284472 1498 raft_consensus.cc:729] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0931967d12847e08d9ff883c85f4237, State: Initialized, Role: FOLLOWER
I20240501 20:08:59.284725 1498 consensus_queue.cc:260] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:08:59.284885 1498 raft_consensus.cc:388] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:59.284979 1498 raft_consensus.cc:482] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:59.285102 1498 raft_consensus.cc:3037] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:59.287588 1498 raft_consensus.cc:504] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:08:59.287848 1498 leader_election.cc:302] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0931967d12847e08d9ff883c85f4237; no voters:
I20240501 20:08:59.288075 1498 leader_election.cc:288] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:59.288157 1500 raft_consensus.cc:2781] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:59.288484 1500 raft_consensus.cc:686] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Becoming Leader. State: Replica: c0931967d12847e08d9ff883c85f4237, State: Running, Role: LEADER
I20240501 20:08:59.288575 1498 ts_tablet_manager.cc:1398] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:08:59.288714 1500 consensus_queue.cc:237] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:08:59.288877 1498 tablet_bootstrap.cc:492] T 76cd9ee2deb14a5aa9bb280b0a8dec53 P c0931967d12847e08d9ff883c85f4237: Bootstrap starting.
I20240501 20:08:59.290766 1498 tablet_bootstrap.cc:654] T 76cd9ee2deb14a5aa9bb280b0a8dec53 P c0931967d12847e08d9ff883c85f4237: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:59.291172 484 catalog_manager.cc:5484] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0931967d12847e08d9ff883c85f4237 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "c0931967d12847e08d9ff883c85f4237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:59.292593 1498 tablet_bootstrap.cc:492] T 76cd9ee2deb14a5aa9bb280b0a8dec53 P c0931967d12847e08d9ff883c85f4237: No bootstrap required, opened a new log
I20240501 20:08:59.292783 1498 ts_tablet_manager.cc:1367] T 76cd9ee2deb14a5aa9bb280b0a8dec53 P c0931967d12847e08d9ff883c85f4237: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 20:08:59.293498 1498 raft_consensus.cc:348] T 76cd9ee2deb14a5aa9bb280b0a8dec53 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:08:59.293696 1498 raft_consensus.cc:374] T 76cd9ee2deb14a5aa9bb280b0a8dec53 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:59.293782 1498 raft_consensus.cc:729] T 76cd9ee2deb14a5aa9bb280b0a8dec53 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0931967d12847e08d9ff883c85f4237, State: Initialized, Role: FOLLOWER
I20240501 20:08:59.294035 1498 consensus_queue.cc:260] T 76cd9ee2deb14a5aa9bb280b0a8dec53 P c0931967d12847e08d9ff883c85f4237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:08:59.294202 1498 raft_consensus.cc:388] T 76cd9ee2deb14a5aa9bb280b0a8dec53 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:59.294322 1498 raft_consensus.cc:482] T 76cd9ee2deb14a5aa9bb280b0a8dec53 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:59.294415 1498 raft_consensus.cc:3037] T 76cd9ee2deb14a5aa9bb280b0a8dec53 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:59.296375 1498 raft_consensus.cc:504] T 76cd9ee2deb14a5aa9bb280b0a8dec53 P c0931967d12847e08d9ff883c85f4237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:08:59.296609 1498 leader_election.cc:302] T 76cd9ee2deb14a5aa9bb280b0a8dec53 P c0931967d12847e08d9ff883c85f4237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0931967d12847e08d9ff883c85f4237; no voters:
I20240501 20:08:59.296809 1498 leader_election.cc:288] T 76cd9ee2deb14a5aa9bb280b0a8dec53 P c0931967d12847e08d9ff883c85f4237 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:59.296903 1500 raft_consensus.cc:2781] T 76cd9ee2deb14a5aa9bb280b0a8dec53 P c0931967d12847e08d9ff883c85f4237 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:59.297089 1500 raft_consensus.cc:686] T 76cd9ee2deb14a5aa9bb280b0a8dec53 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Becoming Leader. State: Replica: c0931967d12847e08d9ff883c85f4237, State: Running, Role: LEADER
I20240501 20:08:59.297288 1498 ts_tablet_manager.cc:1398] T 76cd9ee2deb14a5aa9bb280b0a8dec53 P c0931967d12847e08d9ff883c85f4237: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240501 20:08:59.297315 1500 consensus_queue.cc:237] T 76cd9ee2deb14a5aa9bb280b0a8dec53 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:08:59.299428 483 catalog_manager.cc:5484] T 76cd9ee2deb14a5aa9bb280b0a8dec53 P c0931967d12847e08d9ff883c85f4237 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0931967d12847e08d9ff883c85f4237 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "c0931967d12847e08d9ff883c85f4237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } health_report { overall_health: HEALTHY } } }
W20240501 20:08:59.657295 1509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:08:59.657677 1509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:08:59.665005 1509 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 128.338 ms
time per row: 0.064169 ms
I20240501 20:08:59.935904 478 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57448:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:08:59.945017 736 tablet_service.cc:1467] Processing CreateTablet for tablet 998f81b7cc7e4df7832f4694c4f34731 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c9860c7b5b694a66abe3f7254149aa83]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:59.945132 737 tablet_service.cc:1467] Processing CreateTablet for tablet 5786820dd3d84b86a7a800b034a3e833 (DEFAULT_TABLE table=rebalance_test_table_1 [id=c9860c7b5b694a66abe3f7254149aa83]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:59.945084 735 tablet_service.cc:1467] Processing CreateTablet for tablet 2705dcf15dc54cbbb263fcfee0ee8aed (DEFAULT_TABLE table=rebalance_test_table_1 [id=c9860c7b5b694a66abe3f7254149aa83]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:08:59.945552 736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 998f81b7cc7e4df7832f4694c4f34731. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:59.945742 737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5786820dd3d84b86a7a800b034a3e833. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:59.945974 735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2705dcf15dc54cbbb263fcfee0ee8aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:08:59.952634 1548 tablet_bootstrap.cc:492] T 5786820dd3d84b86a7a800b034a3e833 P b811afcdb9d34a38b7e14e693b58093e: Bootstrap starting.
I20240501 20:08:59.954191 1548 tablet_bootstrap.cc:654] T 5786820dd3d84b86a7a800b034a3e833 P b811afcdb9d34a38b7e14e693b58093e: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:59.956180 1548 tablet_bootstrap.cc:492] T 5786820dd3d84b86a7a800b034a3e833 P b811afcdb9d34a38b7e14e693b58093e: No bootstrap required, opened a new log
I20240501 20:08:59.956305 1548 ts_tablet_manager.cc:1367] T 5786820dd3d84b86a7a800b034a3e833 P b811afcdb9d34a38b7e14e693b58093e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 20:08:59.957055 1548 raft_consensus.cc:348] T 5786820dd3d84b86a7a800b034a3e833 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:08:59.957199 1548 raft_consensus.cc:374] T 5786820dd3d84b86a7a800b034a3e833 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:59.957288 1548 raft_consensus.cc:729] T 5786820dd3d84b86a7a800b034a3e833 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b811afcdb9d34a38b7e14e693b58093e, State: Initialized, Role: FOLLOWER
I20240501 20:08:59.957490 1548 consensus_queue.cc:260] T 5786820dd3d84b86a7a800b034a3e833 P b811afcdb9d34a38b7e14e693b58093e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:08:59.957639 1548 raft_consensus.cc:388] T 5786820dd3d84b86a7a800b034a3e833 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:59.957716 1548 raft_consensus.cc:482] T 5786820dd3d84b86a7a800b034a3e833 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:59.957811 1548 raft_consensus.cc:3037] T 5786820dd3d84b86a7a800b034a3e833 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:59.959815 1548 raft_consensus.cc:504] T 5786820dd3d84b86a7a800b034a3e833 P b811afcdb9d34a38b7e14e693b58093e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:08:59.960062 1548 leader_election.cc:302] T 5786820dd3d84b86a7a800b034a3e833 P b811afcdb9d34a38b7e14e693b58093e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b811afcdb9d34a38b7e14e693b58093e; no voters:
I20240501 20:08:59.960443 1548 leader_election.cc:288] T 5786820dd3d84b86a7a800b034a3e833 P b811afcdb9d34a38b7e14e693b58093e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:59.960613 1550 raft_consensus.cc:2781] T 5786820dd3d84b86a7a800b034a3e833 P b811afcdb9d34a38b7e14e693b58093e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:59.961122 1550 raft_consensus.cc:686] T 5786820dd3d84b86a7a800b034a3e833 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Becoming Leader. State: Replica: b811afcdb9d34a38b7e14e693b58093e, State: Running, Role: LEADER
I20240501 20:08:59.961195 1548 ts_tablet_manager.cc:1398] T 5786820dd3d84b86a7a800b034a3e833 P b811afcdb9d34a38b7e14e693b58093e: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20240501 20:08:59.961531 1548 tablet_bootstrap.cc:492] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e: Bootstrap starting.
I20240501 20:08:59.961510 1550 consensus_queue.cc:237] T 5786820dd3d84b86a7a800b034a3e833 P b811afcdb9d34a38b7e14e693b58093e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:08:59.963186 1548 tablet_bootstrap.cc:654] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:59.964381 478 catalog_manager.cc:5484] T 5786820dd3d84b86a7a800b034a3e833 P b811afcdb9d34a38b7e14e693b58093e reported cstate change: term changed from 0 to 1, leader changed from <none> to b811afcdb9d34a38b7e14e693b58093e (127.29.219.66). New cstate: current_term: 1 leader_uuid: "b811afcdb9d34a38b7e14e693b58093e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:59.965576 1548 tablet_bootstrap.cc:492] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e: No bootstrap required, opened a new log
I20240501 20:08:59.965723 1548 ts_tablet_manager.cc:1367] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:08:59.966301 1548 raft_consensus.cc:348] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:08:59.966499 1548 raft_consensus.cc:374] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:59.966578 1548 raft_consensus.cc:729] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b811afcdb9d34a38b7e14e693b58093e, State: Initialized, Role: FOLLOWER
I20240501 20:08:59.966830 1548 consensus_queue.cc:260] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:08:59.966998 1548 raft_consensus.cc:388] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:59.967095 1548 raft_consensus.cc:482] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:59.967185 1548 raft_consensus.cc:3037] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:59.969377 1548 raft_consensus.cc:504] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:08:59.969581 1548 leader_election.cc:302] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b811afcdb9d34a38b7e14e693b58093e; no voters:
I20240501 20:08:59.969782 1548 leader_election.cc:288] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:59.969851 1550 raft_consensus.cc:2781] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:59.970059 1550 raft_consensus.cc:686] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Becoming Leader. State: Replica: b811afcdb9d34a38b7e14e693b58093e, State: Running, Role: LEADER
I20240501 20:08:59.970320 1548 ts_tablet_manager.cc:1398] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:08:59.970328 1550 consensus_queue.cc:237] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:08:59.970655 1548 tablet_bootstrap.cc:492] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e: Bootstrap starting.
I20240501 20:08:59.972369 1548 tablet_bootstrap.cc:654] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e: Neither blocks nor log segments found. Creating new log.
I20240501 20:08:59.972571 478 catalog_manager.cc:5484] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e reported cstate change: term changed from 0 to 1, leader changed from <none> to b811afcdb9d34a38b7e14e693b58093e (127.29.219.66). New cstate: current_term: 1 leader_uuid: "b811afcdb9d34a38b7e14e693b58093e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } health_report { overall_health: HEALTHY } } }
I20240501 20:08:59.974195 1548 tablet_bootstrap.cc:492] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e: No bootstrap required, opened a new log
I20240501 20:08:59.974325 1548 ts_tablet_manager.cc:1367] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:08:59.975019 1548 raft_consensus.cc:348] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:08:59.975253 1548 raft_consensus.cc:374] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:08:59.975384 1548 raft_consensus.cc:729] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b811afcdb9d34a38b7e14e693b58093e, State: Initialized, Role: FOLLOWER
I20240501 20:08:59.975625 1548 consensus_queue.cc:260] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:08:59.975827 1548 raft_consensus.cc:388] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:08:59.975910 1548 raft_consensus.cc:482] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:08:59.976022 1548 raft_consensus.cc:3037] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:08:59.978499 1548 raft_consensus.cc:504] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:08:59.978761 1548 leader_election.cc:302] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b811afcdb9d34a38b7e14e693b58093e; no voters:
I20240501 20:08:59.978904 1548 leader_election.cc:288] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:08:59.978991 1550 raft_consensus.cc:2781] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:08:59.979180 1550 raft_consensus.cc:686] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Becoming Leader. State: Replica: b811afcdb9d34a38b7e14e693b58093e, State: Running, Role: LEADER
I20240501 20:08:59.979334 1548 ts_tablet_manager.cc:1398] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 20:08:59.979432 1550 consensus_queue.cc:237] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:08:59.981355 478 catalog_manager.cc:5484] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e reported cstate change: term changed from 0 to 1, leader changed from <none> to b811afcdb9d34a38b7e14e693b58093e (127.29.219.66). New cstate: current_term: 1 leader_uuid: "b811afcdb9d34a38b7e14e693b58093e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } health_report { overall_health: HEALTHY } } }
W20240501 20:09:00.325244 1559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:00.325582 1559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:00.333365 1559 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 104.981 ms
time per row: 0.0524904 ms
I20240501 20:09:00.581485 478 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57448:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:00.591657 603 tablet_service.cc:1467] Processing CreateTablet for tablet e4b484eeb8424dbb9844f6b96fdacd34 (DEFAULT_TABLE table=rebalance_test_table_2 [id=318aa2fee23548949c6edf6fe8ff8283]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:00.591778 604 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae809988743421f8046716158b43a88 (DEFAULT_TABLE table=rebalance_test_table_2 [id=318aa2fee23548949c6edf6fe8ff8283]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:00.591657 605 tablet_service.cc:1467] Processing CreateTablet for tablet 09ac60ecdffa40b2a321c810d7890609 (DEFAULT_TABLE table=rebalance_test_table_2 [id=318aa2fee23548949c6edf6fe8ff8283]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:00.592355 603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4b484eeb8424dbb9844f6b96fdacd34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:00.592550 605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09ac60ecdffa40b2a321c810d7890609. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:00.592841 604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae809988743421f8046716158b43a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:00.597540 1595 tablet_bootstrap.cc:492] T e4b484eeb8424dbb9844f6b96fdacd34 P c0931967d12847e08d9ff883c85f4237: Bootstrap starting.
I20240501 20:09:00.599670 1595 tablet_bootstrap.cc:654] T e4b484eeb8424dbb9844f6b96fdacd34 P c0931967d12847e08d9ff883c85f4237: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:00.601823 1595 tablet_bootstrap.cc:492] T e4b484eeb8424dbb9844f6b96fdacd34 P c0931967d12847e08d9ff883c85f4237: No bootstrap required, opened a new log
I20240501 20:09:00.601954 1595 ts_tablet_manager.cc:1367] T e4b484eeb8424dbb9844f6b96fdacd34 P c0931967d12847e08d9ff883c85f4237: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:09:00.602538 1595 raft_consensus.cc:348] T e4b484eeb8424dbb9844f6b96fdacd34 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:09:00.602723 1595 raft_consensus.cc:374] T e4b484eeb8424dbb9844f6b96fdacd34 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:00.602821 1595 raft_consensus.cc:729] T e4b484eeb8424dbb9844f6b96fdacd34 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0931967d12847e08d9ff883c85f4237, State: Initialized, Role: FOLLOWER
I20240501 20:09:00.603044 1595 consensus_queue.cc:260] T e4b484eeb8424dbb9844f6b96fdacd34 P c0931967d12847e08d9ff883c85f4237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:09:00.603193 1595 raft_consensus.cc:388] T e4b484eeb8424dbb9844f6b96fdacd34 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:09:00.603271 1595 raft_consensus.cc:482] T e4b484eeb8424dbb9844f6b96fdacd34 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:09:00.603379 1595 raft_consensus.cc:3037] T e4b484eeb8424dbb9844f6b96fdacd34 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:00.605298 1595 raft_consensus.cc:504] T e4b484eeb8424dbb9844f6b96fdacd34 P c0931967d12847e08d9ff883c85f4237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:09:00.605516 1595 leader_election.cc:302] T e4b484eeb8424dbb9844f6b96fdacd34 P c0931967d12847e08d9ff883c85f4237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0931967d12847e08d9ff883c85f4237; no voters:
I20240501 20:09:00.605887 1595 leader_election.cc:288] T e4b484eeb8424dbb9844f6b96fdacd34 P c0931967d12847e08d9ff883c85f4237 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:09:00.606047 1597 raft_consensus.cc:2781] T e4b484eeb8424dbb9844f6b96fdacd34 P c0931967d12847e08d9ff883c85f4237 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:00.606514 1595 ts_tablet_manager.cc:1398] T e4b484eeb8424dbb9844f6b96fdacd34 P c0931967d12847e08d9ff883c85f4237: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 20:09:00.606693 1597 raft_consensus.cc:686] T e4b484eeb8424dbb9844f6b96fdacd34 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Becoming Leader. State: Replica: c0931967d12847e08d9ff883c85f4237, State: Running, Role: LEADER
I20240501 20:09:00.606832 1595 tablet_bootstrap.cc:492] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237: Bootstrap starting.
I20240501 20:09:00.606967 1597 consensus_queue.cc:237] T e4b484eeb8424dbb9844f6b96fdacd34 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:09:00.608520 1595 tablet_bootstrap.cc:654] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:00.609295 478 catalog_manager.cc:5484] T e4b484eeb8424dbb9844f6b96fdacd34 P c0931967d12847e08d9ff883c85f4237 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0931967d12847e08d9ff883c85f4237 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "c0931967d12847e08d9ff883c85f4237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:00.610561 1595 tablet_bootstrap.cc:492] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237: No bootstrap required, opened a new log
I20240501 20:09:00.610736 1595 ts_tablet_manager.cc:1367] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 20:09:00.611703 1595 raft_consensus.cc:348] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:09:00.611920 1595 raft_consensus.cc:374] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:00.612015 1595 raft_consensus.cc:729] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0931967d12847e08d9ff883c85f4237, State: Initialized, Role: FOLLOWER
I20240501 20:09:00.612243 1595 consensus_queue.cc:260] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:09:00.612390 1595 raft_consensus.cc:388] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:09:00.612496 1595 raft_consensus.cc:482] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:09:00.612591 1595 raft_consensus.cc:3037] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:00.614902 1595 raft_consensus.cc:504] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:09:00.615109 1595 leader_election.cc:302] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0931967d12847e08d9ff883c85f4237; no voters:
I20240501 20:09:00.615306 1595 leader_election.cc:288] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:09:00.615351 1597 raft_consensus.cc:2781] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:00.615554 1597 raft_consensus.cc:686] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Becoming Leader. State: Replica: c0931967d12847e08d9ff883c85f4237, State: Running, Role: LEADER
I20240501 20:09:00.615779 1595 ts_tablet_manager.cc:1398] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240501 20:09:00.615813 1597 consensus_queue.cc:237] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:09:00.616062 1595 tablet_bootstrap.cc:492] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237: Bootstrap starting.
I20240501 20:09:00.617636 1595 tablet_bootstrap.cc:654] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:00.617959 478 catalog_manager.cc:5484] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0931967d12847e08d9ff883c85f4237 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "c0931967d12847e08d9ff883c85f4237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:00.619388 1595 tablet_bootstrap.cc:492] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237: No bootstrap required, opened a new log
I20240501 20:09:00.619561 1595 ts_tablet_manager.cc:1367] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240501 20:09:00.620182 1595 raft_consensus.cc:348] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:09:00.620337 1595 raft_consensus.cc:374] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:00.620430 1595 raft_consensus.cc:729] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0931967d12847e08d9ff883c85f4237, State: Initialized, Role: FOLLOWER
I20240501 20:09:00.620689 1595 consensus_queue.cc:260] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:09:00.620908 1595 raft_consensus.cc:388] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:09:00.620991 1595 raft_consensus.cc:482] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:09:00.621106 1595 raft_consensus.cc:3037] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:00.623572 1595 raft_consensus.cc:504] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:09:00.623811 1595 leader_election.cc:302] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0931967d12847e08d9ff883c85f4237; no voters:
I20240501 20:09:00.623978 1595 leader_election.cc:288] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:09:00.624060 1597 raft_consensus.cc:2781] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:00.624213 1597 raft_consensus.cc:686] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Becoming Leader. State: Replica: c0931967d12847e08d9ff883c85f4237, State: Running, Role: LEADER
I20240501 20:09:00.624431 1595 ts_tablet_manager.cc:1398] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240501 20:09:00.624457 1597 consensus_queue.cc:237] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:09:00.626605 478 catalog_manager.cc:5484] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0931967d12847e08d9ff883c85f4237 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "c0931967d12847e08d9ff883c85f4237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } health_report { overall_health: HEALTHY } } }
W20240501 20:09:00.963701 1606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:00.964015 1606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:00.971547 1606 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 132.22 ms
time per row: 0.0661102 ms
I20240501 20:09:01.249050 478 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57448:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:01.257637 735 tablet_service.cc:1467] Processing CreateTablet for tablet e606598eee444113b75b26943d884754 (DEFAULT_TABLE table=rebalance_test_table_3 [id=54c7219e95dd4c53bfd6d508026d4771]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:01.257792 736 tablet_service.cc:1467] Processing CreateTablet for tablet 0a9d19f112154a65b012072f9d0d4149 (DEFAULT_TABLE table=rebalance_test_table_3 [id=54c7219e95dd4c53bfd6d508026d4771]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:01.257884 737 tablet_service.cc:1467] Processing CreateTablet for tablet 1b76d484a0a7446081d3e7dde20af484 (DEFAULT_TABLE table=rebalance_test_table_3 [id=54c7219e95dd4c53bfd6d508026d4771]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:01.258138 735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e606598eee444113b75b26943d884754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:01.258342 737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b76d484a0a7446081d3e7dde20af484. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:01.258559 736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a9d19f112154a65b012072f9d0d4149. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:01.267653 1644 tablet_bootstrap.cc:492] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e: Bootstrap starting.
I20240501 20:09:01.269694 1644 tablet_bootstrap.cc:654] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:01.271720 1644 tablet_bootstrap.cc:492] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e: No bootstrap required, opened a new log
I20240501 20:09:01.271864 1644 ts_tablet_manager.cc:1367] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240501 20:09:01.272612 1644 raft_consensus.cc:348] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:09:01.272789 1644 raft_consensus.cc:374] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:01.272889 1644 raft_consensus.cc:729] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b811afcdb9d34a38b7e14e693b58093e, State: Initialized, Role: FOLLOWER
I20240501 20:09:01.273154 1644 consensus_queue.cc:260] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:09:01.273344 1644 raft_consensus.cc:388] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:09:01.273420 1644 raft_consensus.cc:482] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:09:01.273520 1644 raft_consensus.cc:3037] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:01.276394 1644 raft_consensus.cc:504] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:09:01.276770 1644 leader_election.cc:302] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b811afcdb9d34a38b7e14e693b58093e; no voters:
I20240501 20:09:01.277287 1644 leader_election.cc:288] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:09:01.277453 1646 raft_consensus.cc:2781] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:01.278038 1644 ts_tablet_manager.cc:1398] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20240501 20:09:01.278302 1644 tablet_bootstrap.cc:492] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e: Bootstrap starting.
I20240501 20:09:01.280022 1644 tablet_bootstrap.cc:654] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:01.282115 1644 tablet_bootstrap.cc:492] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e: No bootstrap required, opened a new log
I20240501 20:09:01.282279 1644 ts_tablet_manager.cc:1367] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:09:01.283075 1644 raft_consensus.cc:348] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:09:01.283258 1644 raft_consensus.cc:374] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:01.283329 1644 raft_consensus.cc:729] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b811afcdb9d34a38b7e14e693b58093e, State: Initialized, Role: FOLLOWER
I20240501 20:09:01.283576 1644 consensus_queue.cc:260] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:09:01.283778 1644 raft_consensus.cc:388] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:09:01.283877 1644 raft_consensus.cc:482] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:09:01.284080 1644 raft_consensus.cc:3037] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:01.288005 1644 raft_consensus.cc:504] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:09:01.288287 1644 leader_election.cc:302] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b811afcdb9d34a38b7e14e693b58093e; no voters:
I20240501 20:09:01.288506 1644 leader_election.cc:288] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:09:01.288643 1647 raft_consensus.cc:2781] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:01.288975 1644 ts_tablet_manager.cc:1398] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240501 20:09:01.289299 1644 tablet_bootstrap.cc:492] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e: Bootstrap starting.
I20240501 20:09:01.290171 1646 raft_consensus.cc:686] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Becoming Leader. State: Replica: b811afcdb9d34a38b7e14e693b58093e, State: Running, Role: LEADER
I20240501 20:09:01.290589 1647 raft_consensus.cc:686] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Becoming Leader. State: Replica: b811afcdb9d34a38b7e14e693b58093e, State: Running, Role: LEADER
I20240501 20:09:01.290561 1646 consensus_queue.cc:237] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:09:01.290858 1647 consensus_queue.cc:237] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:09:01.291384 1644 tablet_bootstrap.cc:654] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:01.293594 478 catalog_manager.cc:5484] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e reported cstate change: term changed from 0 to 1, leader changed from <none> to b811afcdb9d34a38b7e14e693b58093e (127.29.219.66). New cstate: current_term: 1 leader_uuid: "b811afcdb9d34a38b7e14e693b58093e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:01.300474 1644 tablet_bootstrap.cc:492] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e: No bootstrap required, opened a new log
I20240501 20:09:01.300624 1644 ts_tablet_manager.cc:1367] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240501 20:09:01.301187 1644 raft_consensus.cc:348] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:09:01.301329 1644 raft_consensus.cc:374] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:01.301218 478 catalog_manager.cc:5484] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e reported cstate change: term changed from 0 to 1, leader changed from <none> to b811afcdb9d34a38b7e14e693b58093e (127.29.219.66). New cstate: current_term: 1 leader_uuid: "b811afcdb9d34a38b7e14e693b58093e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:01.301422 1644 raft_consensus.cc:729] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b811afcdb9d34a38b7e14e693b58093e, State: Initialized, Role: FOLLOWER
I20240501 20:09:01.301656 1644 consensus_queue.cc:260] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:09:01.301823 1644 raft_consensus.cc:388] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:09:01.301891 1644 raft_consensus.cc:482] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:09:01.302021 1644 raft_consensus.cc:3037] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:01.304358 1644 raft_consensus.cc:504] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:09:01.304596 1644 leader_election.cc:302] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b811afcdb9d34a38b7e14e693b58093e; no voters:
I20240501 20:09:01.304738 1644 leader_election.cc:288] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:09:01.304854 1650 raft_consensus.cc:2781] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:01.304981 1650 raft_consensus.cc:686] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Becoming Leader. State: Replica: b811afcdb9d34a38b7e14e693b58093e, State: Running, Role: LEADER
I20240501 20:09:01.305207 1644 ts_tablet_manager.cc:1398] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:09:01.305253 1650 consensus_queue.cc:237] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } }
I20240501 20:09:01.307227 478 catalog_manager.cc:5484] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e reported cstate change: term changed from 0 to 1, leader changed from <none> to b811afcdb9d34a38b7e14e693b58093e (127.29.219.66). New cstate: current_term: 1 leader_uuid: "b811afcdb9d34a38b7e14e693b58093e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } health_report { overall_health: HEALTHY } } }
W20240501 20:09:01.648533 1657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:01.648877 1657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:01.656679 1657 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 116.403 ms
time per row: 0.0582013 ms
I20240501 20:09:01.915277 483 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57448:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:01.923784 605 tablet_service.cc:1467] Processing CreateTablet for tablet 6cee76b4a3ef4e7a87714c210cf705ac (DEFAULT_TABLE table=rebalance_test_table_4 [id=e27501566faf443498994ced607143d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:01.923784 604 tablet_service.cc:1467] Processing CreateTablet for tablet d2f958611c6c4f459055bd2a7cc75f15 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e27501566faf443498994ced607143d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:01.923970 603 tablet_service.cc:1467] Processing CreateTablet for tablet f71d46683ee74f0b8adb7ffdc4ec2688 (DEFAULT_TABLE table=rebalance_test_table_4 [id=e27501566faf443498994ced607143d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:01.924319 605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cee76b4a3ef4e7a87714c210cf705ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:01.924500 603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71d46683ee74f0b8adb7ffdc4ec2688. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:01.924695 604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2f958611c6c4f459055bd2a7cc75f15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:01.930666 1694 tablet_bootstrap.cc:492] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237: Bootstrap starting.
I20240501 20:09:01.932116 1694 tablet_bootstrap.cc:654] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:01.933761 1694 tablet_bootstrap.cc:492] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237: No bootstrap required, opened a new log
I20240501 20:09:01.933880 1694 ts_tablet_manager.cc:1367] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:01.934453 1694 raft_consensus.cc:348] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:09:01.934594 1694 raft_consensus.cc:374] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:01.934723 1694 raft_consensus.cc:729] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0931967d12847e08d9ff883c85f4237, State: Initialized, Role: FOLLOWER
I20240501 20:09:01.934955 1694 consensus_queue.cc:260] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:09:01.935099 1694 raft_consensus.cc:388] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:09:01.935169 1694 raft_consensus.cc:482] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:09:01.935273 1694 raft_consensus.cc:3037] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:01.937104 1694 raft_consensus.cc:504] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:09:01.937326 1694 leader_election.cc:302] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0931967d12847e08d9ff883c85f4237; no voters:
I20240501 20:09:01.937775 1694 leader_election.cc:288] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:09:01.937938 1696 raft_consensus.cc:2781] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:01.938472 1696 raft_consensus.cc:686] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Becoming Leader. State: Replica: c0931967d12847e08d9ff883c85f4237, State: Running, Role: LEADER
I20240501 20:09:01.938746 1696 consensus_queue.cc:237] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:09:01.938818 1694 ts_tablet_manager.cc:1398] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20240501 20:09:01.939036 1694 tablet_bootstrap.cc:492] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237: Bootstrap starting.
I20240501 20:09:01.940776 1694 tablet_bootstrap.cc:654] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:01.942848 1694 tablet_bootstrap.cc:492] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237: No bootstrap required, opened a new log
I20240501 20:09:01.943017 1694 ts_tablet_manager.cc:1367] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240501 20:09:01.943442 483 catalog_manager.cc:5484] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0931967d12847e08d9ff883c85f4237 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "c0931967d12847e08d9ff883c85f4237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:01.943679 1694 raft_consensus.cc:348] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:09:01.943820 1694 raft_consensus.cc:374] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:01.943872 1694 raft_consensus.cc:729] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0931967d12847e08d9ff883c85f4237, State: Initialized, Role: FOLLOWER
I20240501 20:09:01.944065 1694 consensus_queue.cc:260] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:09:01.944183 1694 raft_consensus.cc:388] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:09:01.944263 1694 raft_consensus.cc:482] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:09:01.944365 1694 raft_consensus.cc:3037] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:01.946401 1694 raft_consensus.cc:504] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:09:01.946743 1694 leader_election.cc:302] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0931967d12847e08d9ff883c85f4237; no voters:
I20240501 20:09:01.946985 1694 leader_election.cc:288] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:09:01.947023 1696 raft_consensus.cc:2781] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:01.947227 1696 raft_consensus.cc:686] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Becoming Leader. State: Replica: c0931967d12847e08d9ff883c85f4237, State: Running, Role: LEADER
I20240501 20:09:01.947470 1694 ts_tablet_manager.cc:1398] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20240501 20:09:01.947479 1696 consensus_queue.cc:237] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:09:01.947786 1694 tablet_bootstrap.cc:492] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237: Bootstrap starting.
I20240501 20:09:01.949178 1694 tablet_bootstrap.cc:654] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:01.951462 1694 tablet_bootstrap.cc:492] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237: No bootstrap required, opened a new log
I20240501 20:09:01.951864 1694 ts_tablet_manager.cc:1367] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20240501 20:09:01.952564 1694 raft_consensus.cc:348] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:09:01.952522 483 catalog_manager.cc:5484] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0931967d12847e08d9ff883c85f4237 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "c0931967d12847e08d9ff883c85f4237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:01.952728 1694 raft_consensus.cc:374] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:01.952811 1694 raft_consensus.cc:729] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0931967d12847e08d9ff883c85f4237, State: Initialized, Role: FOLLOWER
I20240501 20:09:01.953030 1694 consensus_queue.cc:260] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:09:01.953222 1694 raft_consensus.cc:388] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:09:01.953305 1694 raft_consensus.cc:482] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:09:01.953414 1694 raft_consensus.cc:3037] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:01.955673 1694 raft_consensus.cc:504] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:09:01.955902 1694 leader_election.cc:302] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0931967d12847e08d9ff883c85f4237; no voters:
I20240501 20:09:01.956085 1694 leader_election.cc:288] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:09:01.956174 1696 raft_consensus.cc:2781] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:01.956383 1696 raft_consensus.cc:686] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Becoming Leader. State: Replica: c0931967d12847e08d9ff883c85f4237, State: Running, Role: LEADER
I20240501 20:09:01.956590 1694 ts_tablet_manager.cc:1398] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:09:01.956617 1696 consensus_queue.cc:237] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } }
I20240501 20:09:01.958527 483 catalog_manager.cc:5484] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0931967d12847e08d9ff883c85f4237 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "c0931967d12847e08d9ff883c85f4237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } health_report { overall_health: HEALTHY } } }
W20240501 20:09:02.301155 1705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:02.301470 1705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:02.309149 1705 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 124.546 ms
time per row: 0.0622728 ms
I20240501 20:09:02.575316 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.67:42219
--local_ip_for_outbound_sockets=127.29.219.67
--tserver_master_addrs=127.29.219.126:46839
--webserver_port=36873
--webserver_interface=127.29.219.67
--builtin_ntp_servers=127.29.219.84:42611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:09:02.904457 1739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:02.904760 1739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:02.904850 1739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:02.912511 1739 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:02.912645 1739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:02.912792 1739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.67
I20240501 20:09:02.922292 1739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:42611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.67:42219
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.29.219.67
--webserver_port=36873
--tserver_master_addrs=127.29.219.126:46839
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.67
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:02.923893 1739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:02.925670 1739 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:02.938102 1744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:02.939216 1747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:02.939886 1745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:02.941182 1739 server_base.cc:1034] running on GCE node
I20240501 20:09:02.941861 1739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:02.942912 1739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:02.944159 1739 hybrid_clock.cc:648] HybridClock initialized: now 1714594142944113 us; error 38 us; skew 500 ppm
I20240501 20:09:02.944463 1739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:02.946700 1739 webserver.cc:460] Webserver started at http://127.29.219.67:36873/ using document root <none> and password file <none>
I20240501 20:09:02.947520 1739 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:02.947629 1739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:02.952824 1739 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20240501 20:09:02.955636 1753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:02.957063 1739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 20:09:02.957197 1739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "be64c56dce3d46fa80ba75efff719a7d"
format_stamp: "Formatted at 2024-05-01 20:08:52 on dist-test-slave-h2cj"
I20240501 20:09:02.957993 1739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:02.986979 1739 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:02.987883 1739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:02.988168 1739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:02.989089 1739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:02.990597 1739 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:02.990710 1739 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:02.990811 1739 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:02.990854 1739 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:03.021930 1739 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.67:42219
I20240501 20:09:03.021992 1866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.67:42219 every 8 connection(s)
I20240501 20:09:03.023496 1739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:09:03.024530 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 1739
I20240501 20:09:03.025285 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.68:46431
--local_ip_for_outbound_sockets=127.29.219.68
--tserver_master_addrs=127.29.219.126:46839
--webserver_port=34497
--webserver_interface=127.29.219.68
--builtin_ntp_servers=127.29.219.84:42611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:03.038403 1867 heartbeater.cc:347] Connected to a master server at 127.29.219.126:46839
I20240501 20:09:03.038808 1867 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:03.039944 1867 heartbeater.cc:511] Master 127.29.219.126:46839 requested a full tablet report, sending...
I20240501 20:09:03.041950 483 ts_manager.cc:194] Re-registered known tserver with Master: be64c56dce3d46fa80ba75efff719a7d (127.29.219.67:42219)
I20240501 20:09:03.043318 483 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.67:51183
W20240501 20:09:03.367996 1871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:03.368305 1871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:03.368440 1871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:03.376195 1871 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:03.376327 1871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:03.376439 1871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.68
I20240501 20:09:03.385568 1871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:42611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.68:46431
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.29.219.68
--webserver_port=34497
--tserver_master_addrs=127.29.219.126:46839
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.68
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:03.386874 1871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:03.388582 1871 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:03.401062 1877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:03.402079 1876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:03.403077 1879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:03.404271 1871 server_base.cc:1034] running on GCE node
I20240501 20:09:03.404942 1871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:03.405941 1871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:03.407171 1871 hybrid_clock.cc:648] HybridClock initialized: now 1714594143407112 us; error 45 us; skew 500 ppm
I20240501 20:09:03.407529 1871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:03.410012 1871 webserver.cc:460] Webserver started at http://127.29.219.68:34497/ using document root <none> and password file <none>
I20240501 20:09:03.410831 1871 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:03.410969 1871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:03.416455 1871 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20240501 20:09:03.419489 1885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:03.421036 1871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:03.421228 1871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "52541d349ee24edaac72825773fc848a"
format_stamp: "Formatted at 2024-05-01 20:08:53 on dist-test-slave-h2cj"
I20240501 20:09:03.422160 1871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:03.451534 1871 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:03.452409 1871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:03.452694 1871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:03.453663 1871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:03.455395 1871 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:03.455471 1871 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:03.455554 1871 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:03.455597 1871 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:03.486469 1871 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.68:46431
I20240501 20:09:03.486521 1998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.68:46431 every 8 connection(s)
I20240501 20:09:03.487931 1871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:09:03.497632 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 1871
I20240501 20:09:03.498418 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.69:44913
--local_ip_for_outbound_sockets=127.29.219.69
--tserver_master_addrs=127.29.219.126:46839
--webserver_port=38001
--webserver_interface=127.29.219.69
--builtin_ntp_servers=127.29.219.84:42611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:03.500602 1999 heartbeater.cc:347] Connected to a master server at 127.29.219.126:46839
I20240501 20:09:03.500916 1999 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:03.501758 1999 heartbeater.cc:511] Master 127.29.219.126:46839 requested a full tablet report, sending...
I20240501 20:09:03.503197 483 ts_manager.cc:194] Re-registered known tserver with Master: 52541d349ee24edaac72825773fc848a (127.29.219.68:46431)
I20240501 20:09:03.504014 483 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.68:44941
W20240501 20:09:03.829135 2002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:03.829449 2002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:03.829591 2002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:03.837134 2002 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:03.837265 2002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:03.837426 2002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.69
I20240501 20:09:03.846756 2002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:42611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.69:44913
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.29.219.69
--webserver_port=38001
--tserver_master_addrs=127.29.219.126:46839
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.69
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:03.848146 2002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:03.849916 2002 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:03.861642 2008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:03.862545 2007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:03.863585 2010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:03.864516 2002 server_base.cc:1034] running on GCE node
I20240501 20:09:03.865221 2002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:03.866299 2002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:03.867535 2002 hybrid_clock.cc:648] HybridClock initialized: now 1714594143867477 us; error 45 us; skew 500 ppm
I20240501 20:09:03.867980 2002 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:03.874441 2002 webserver.cc:460] Webserver started at http://127.29.219.69:38001/ using document root <none> and password file <none>
I20240501 20:09:03.875279 2002 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:03.875424 2002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:03.880831 2002 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20240501 20:09:03.883759 2016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:03.885074 2002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 20:09:03.885242 2002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "efa05351dbe344c7a451e7fdbeed47f0"
format_stamp: "Formatted at 2024-05-01 20:08:53 on dist-test-slave-h2cj"
I20240501 20:09:03.886065 2002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:03.906292 2002 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:03.907234 2002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:03.907517 2002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:03.908461 2002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:03.909981 2002 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:03.910059 2002 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:03.910187 2002 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:03.910241 2002 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:03.942678 2002 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.69:44913
I20240501 20:09:03.942734 2128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.69:44913 every 8 connection(s)
I20240501 20:09:03.944126 2002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:09:03.947889 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 2002
I20240501 20:09:03.948640 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.70:40955
--local_ip_for_outbound_sockets=127.29.219.70
--tserver_master_addrs=127.29.219.126:46839
--webserver_port=39483
--webserver_interface=127.29.219.70
--builtin_ntp_servers=127.29.219.84:42611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:03.958458 2129 heartbeater.cc:347] Connected to a master server at 127.29.219.126:46839
I20240501 20:09:03.958854 2129 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:03.959676 2129 heartbeater.cc:511] Master 127.29.219.126:46839 requested a full tablet report, sending...
I20240501 20:09:03.961134 483 ts_manager.cc:194] Re-registered known tserver with Master: efa05351dbe344c7a451e7fdbeed47f0 (127.29.219.69:44913)
I20240501 20:09:03.962052 483 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.69:33985
I20240501 20:09:04.046815 1867 heartbeater.cc:503] Master 127.29.219.126:46839 was elected leader, sending a full tablet report...
W20240501 20:09:04.292675 2132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:04.292958 2132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:04.293129 2132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:04.300529 2132 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:04.300690 2132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:04.300817 2132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.70
I20240501 20:09:04.309875 2132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:42611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.70:40955
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.29.219.70
--webserver_port=39483
--tserver_master_addrs=127.29.219.126:46839
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.70
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:04.311215 2132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:04.312901 2132 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:04.325551 2137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:04.326596 2138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:04.327471 2140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:04.328610 2132 server_base.cc:1034] running on GCE node
I20240501 20:09:04.329257 2132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:04.330233 2132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:04.331427 2132 hybrid_clock.cc:648] HybridClock initialized: now 1714594144331373 us; error 58 us; skew 500 ppm
I20240501 20:09:04.331743 2132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:04.334093 2132 webserver.cc:460] Webserver started at http://127.29.219.70:39483/ using document root <none> and password file <none>
I20240501 20:09:04.334883 2132 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:04.335016 2132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:04.340512 2132 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20240501 20:09:04.343307 2146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:04.344614 2132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:04.344789 2132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "09cd35a57c9d4c0f86544e6ac1a55279"
format_stamp: "Formatted at 2024-05-01 20:08:54 on dist-test-slave-h2cj"
I20240501 20:09:04.345611 2132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:04.362260 2132 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:04.363200 2132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:04.363500 2132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:04.364425 2132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:04.365983 2132 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:04.366065 2132 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:04.366199 2132 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:04.366252 2132 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:04.397343 2132 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.70:40955
I20240501 20:09:04.397392 2259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.70:40955 every 8 connection(s)
I20240501 20:09:04.398941 2132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:09:04.400322 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 2132
I20240501 20:09:04.401014 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.71:39829
--local_ip_for_outbound_sockets=127.29.219.71
--tserver_master_addrs=127.29.219.126:46839
--webserver_port=40055
--webserver_interface=127.29.219.71
--builtin_ntp_servers=127.29.219.84:42611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:04.411965 2260 heartbeater.cc:347] Connected to a master server at 127.29.219.126:46839
I20240501 20:09:04.412323 2260 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:04.413151 2260 heartbeater.cc:511] Master 127.29.219.126:46839 requested a full tablet report, sending...
I20240501 20:09:04.414669 483 ts_manager.cc:194] Re-registered known tserver with Master: 09cd35a57c9d4c0f86544e6ac1a55279 (127.29.219.70:40955)
I20240501 20:09:04.415452 483 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.70:46137
I20240501 20:09:04.506531 1999 heartbeater.cc:503] Master 127.29.219.126:46839 was elected leader, sending a full tablet report...
W20240501 20:09:04.734966 2263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:04.735261 2263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:04.735348 2263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:04.743132 2263 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:04.743257 2263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:04.743376 2263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.71
I20240501 20:09:04.752497 2263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:42611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.71:39829
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.29.219.71
--webserver_port=40055
--tserver_master_addrs=127.29.219.126:46839
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.71
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:04.753803 2263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:04.755538 2263 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:04.767372 2269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:04.768493 2268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:04.769585 2271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:04.770515 2263 server_base.cc:1034] running on GCE node
I20240501 20:09:04.771178 2263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:04.772145 2263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:04.773372 2263 hybrid_clock.cc:648] HybridClock initialized: now 1714594144773314 us; error 48 us; skew 500 ppm
I20240501 20:09:04.773742 2263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:04.776487 2263 webserver.cc:460] Webserver started at http://127.29.219.71:40055/ using document root <none> and password file <none>
I20240501 20:09:04.777400 2263 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:04.777540 2263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:04.783372 2263 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20240501 20:09:04.786199 2277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:04.787653 2263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:04.787811 2263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "52675e825f834cf3ac80a5777be65973"
format_stamp: "Formatted at 2024-05-01 20:08:54 on dist-test-slave-h2cj"
I20240501 20:09:04.788626 2263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:04.808821 2263 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:04.809599 2263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:04.809891 2263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:04.810812 2263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:04.812276 2263 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:04.812352 2263 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:04.812470 2263 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:04.812520 2263 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:04.843439 2263 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.71:39829
I20240501 20:09:04.843472 2391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.71:39829 every 8 connection(s)
I20240501 20:09:04.844877 2263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 20:09:04.850783 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 2263
I20240501 20:09:04.858153 2392 heartbeater.cc:347] Connected to a master server at 127.29.219.126:46839
I20240501 20:09:04.858458 2392 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:04.859381 2392 heartbeater.cc:511] Master 127.29.219.126:46839 requested a full tablet report, sending...
I20240501 20:09:04.860782 483 ts_manager.cc:194] Re-registered known tserver with Master: 52675e825f834cf3ac80a5777be65973 (127.29.219.71:39829)
I20240501 20:09:04.861546 483 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.71:58867
I20240501 20:09:04.964809 2129 heartbeater.cc:503] Master 127.29.219.126:46839 was elected leader, sending a full tablet report...
I20240501 20:09:05.402649 1933 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:05.413398 2195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:05.415263 737 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:09:05.419351 2260 heartbeater.cc:503] Master 127.29.219.126:46839 was elected leader, sending a full tablet report...
I20240501 20:09:05.422406 2326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:05.432945 1802 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:05.448079 604 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:05.496587 2064 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:05.672561 2195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:05.675081 1933 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:05.692188 1802 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:05.695319 2326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:05.723666 604 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:09:05.726364 737 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:09:05.748600 2064 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:05.828045 758 consensus_queue.cc:237] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: true } }
I20240501 20:09:05.830736 2482 raft_consensus.cc:2932] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 52675e825f834cf3ac80a5777be65973 (127.29.219.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: true } } }
I20240501 20:09:05.837083 482 catalog_manager.cc:5484] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e reported cstate change: config changed from index -1 to 4, NON_VOTER 52675e825f834cf3ac80a5777be65973 (127.29.219.71) added. New cstate: current_term: 1 leader_uuid: "b811afcdb9d34a38b7e14e693b58093e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:05.841352 690 consensus_peers.cc:485] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e -> Peer 52675e825f834cf3ac80a5777be65973 (127.29.219.71:39829): Couldn't send request to peer 52675e825f834cf3ac80a5777be65973. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a9d19f112154a65b012072f9d0d4149. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:05.852669 625 consensus_queue.cc:237] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: true } }
I20240501 20:09:05.855926 2495 raft_consensus.cc:2932] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER efa05351dbe344c7a451e7fdbeed47f0 (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: true } } }
I20240501 20:09:05.863978 482 catalog_manager.cc:5484] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 reported cstate change: config changed from index -1 to 4, NON_VOTER efa05351dbe344c7a451e7fdbeed47f0 (127.29.219.69) added. New cstate: current_term: 1 leader_uuid: "c0931967d12847e08d9ff883c85f4237" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:05.864400 2392 heartbeater.cc:503] Master 127.29.219.126:46839 was elected leader, sending a full tablet report...
W20240501 20:09:05.869756 558 consensus_peers.cc:485] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 -> Peer efa05351dbe344c7a451e7fdbeed47f0 (127.29.219.69:44913): Couldn't send request to peer efa05351dbe344c7a451e7fdbeed47f0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ad6ed0144eb4b619964ee4f86b9a842. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:05.877434 625 consensus_queue.cc:237] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: true } }
I20240501 20:09:05.879472 2495 raft_consensus.cc:2932] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be64c56dce3d46fa80ba75efff719a7d (127.29.219.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: true } } }
I20240501 20:09:05.885252 482 catalog_manager.cc:5484] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 reported cstate change: config changed from index -1 to 4, NON_VOTER be64c56dce3d46fa80ba75efff719a7d (127.29.219.67) added. New cstate: current_term: 1 leader_uuid: "c0931967d12847e08d9ff883c85f4237" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:05.888177 559 consensus_peers.cc:485] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 -> Peer be64c56dce3d46fa80ba75efff719a7d (127.29.219.67:42219): Couldn't send request to peer be64c56dce3d46fa80ba75efff719a7d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a1f02449b91442a897ac2278ee98e29. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:05.897383 758 consensus_queue.cc:237] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: true } }
I20240501 20:09:05.900033 2481 raft_consensus.cc:2932] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 09cd35a57c9d4c0f86544e6ac1a55279 (127.29.219.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: true } } }
I20240501 20:09:05.908931 482 catalog_manager.cc:5484] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e reported cstate change: config changed from index -1 to 4, NON_VOTER 09cd35a57c9d4c0f86544e6ac1a55279 (127.29.219.70) added. New cstate: current_term: 1 leader_uuid: "b811afcdb9d34a38b7e14e693b58093e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:05.916225 692 consensus_peers.cc:485] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e -> Peer 09cd35a57c9d4c0f86544e6ac1a55279 (127.29.219.70:40955): Couldn't send request to peer 09cd35a57c9d4c0f86544e6ac1a55279. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e606598eee444113b75b26943d884754. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:05.923565 625 consensus_queue.cc:237] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: true } }
I20240501 20:09:05.925604 2495 raft_consensus.cc:2932] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 52541d349ee24edaac72825773fc848a (127.29.219.68) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: true } } }
I20240501 20:09:05.931583 482 catalog_manager.cc:5484] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 reported cstate change: config changed from index -1 to 4, NON_VOTER 52541d349ee24edaac72825773fc848a (127.29.219.68) added. New cstate: current_term: 1 leader_uuid: "c0931967d12847e08d9ff883c85f4237" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:05.934718 561 consensus_peers.cc:485] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 -> Peer 52541d349ee24edaac72825773fc848a (127.29.219.68:46431): Couldn't send request to peer 52541d349ee24edaac72825773fc848a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09ac60ecdffa40b2a321c810d7890609. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:05.944312 625 consensus_queue.cc:237] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: true } }
I20240501 20:09:05.946319 2495 raft_consensus.cc:2932] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 52675e825f834cf3ac80a5777be65973 (127.29.219.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: true } } }
I20240501 20:09:05.952574 482 catalog_manager.cc:5484] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 reported cstate change: config changed from index -1 to 4, NON_VOTER 52675e825f834cf3ac80a5777be65973 (127.29.219.71) added. New cstate: current_term: 1 leader_uuid: "c0931967d12847e08d9ff883c85f4237" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:05.954836 558 consensus_peers.cc:485] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 -> Peer 52675e825f834cf3ac80a5777be65973 (127.29.219.71:39829): Couldn't send request to peer 52675e825f834cf3ac80a5777be65973. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ae809988743421f8046716158b43a88. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:05.965104 758 consensus_queue.cc:237] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: true } }
I20240501 20:09:05.967028 2482 raft_consensus.cc:2932] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER efa05351dbe344c7a451e7fdbeed47f0 (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: true } } }
I20240501 20:09:05.972734 482 catalog_manager.cc:5484] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e reported cstate change: config changed from index -1 to 4, NON_VOTER efa05351dbe344c7a451e7fdbeed47f0 (127.29.219.69) added. New cstate: current_term: 1 leader_uuid: "b811afcdb9d34a38b7e14e693b58093e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:05.975514 690 consensus_peers.cc:485] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e -> Peer efa05351dbe344c7a451e7fdbeed47f0 (127.29.219.69:44913): Couldn't send request to peer efa05351dbe344c7a451e7fdbeed47f0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2705dcf15dc54cbbb263fcfee0ee8aed. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:05.985366 625 consensus_queue.cc:237] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: true } }
I20240501 20:09:05.987326 2495 raft_consensus.cc:2932] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER be64c56dce3d46fa80ba75efff719a7d (127.29.219.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: true } } }
W20240501 20:09:05.987694 559 consensus_peers.cc:485] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 -> Peer be64c56dce3d46fa80ba75efff719a7d (127.29.219.67:42219): Couldn't send request to peer be64c56dce3d46fa80ba75efff719a7d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f71d46683ee74f0b8adb7ffdc4ec2688. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:05.991958 482 catalog_manager.cc:5484] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 reported cstate change: config changed from index -1 to 4, NON_VOTER be64c56dce3d46fa80ba75efff719a7d (127.29.219.67) added. New cstate: current_term: 1 leader_uuid: "c0931967d12847e08d9ff883c85f4237" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:06.003111 758 consensus_queue.cc:237] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: true } }
I20240501 20:09:06.005131 2481 raft_consensus.cc:2932] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 52541d349ee24edaac72825773fc848a (127.29.219.68) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: true } } }
I20240501 20:09:06.010536 482 catalog_manager.cc:5484] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e reported cstate change: config changed from index -1 to 4, NON_VOTER 52541d349ee24edaac72825773fc848a (127.29.219.68) added. New cstate: current_term: 1 leader_uuid: "b811afcdb9d34a38b7e14e693b58093e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:06.013808 693 consensus_peers.cc:485] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e -> Peer 52541d349ee24edaac72825773fc848a (127.29.219.68:46431): Couldn't send request to peer 52541d349ee24edaac72825773fc848a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 998f81b7cc7e4df7832f4694c4f34731. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:06.336252 2597 ts_tablet_manager.cc:902] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a: Initiating tablet copy from peer c0931967d12847e08d9ff883c85f4237 (127.29.219.65:36897)
I20240501 20:09:06.340549 2597 tablet_copy_client.cc:308] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.65:36897
I20240501 20:09:06.349984 645 tablet_copy_service.cc:140] P c0931967d12847e08d9ff883c85f4237: Received BeginTabletCopySession request for tablet 09ac60ecdffa40b2a321c810d7890609 from peer 52541d349ee24edaac72825773fc848a ({username='slave'} at 127.29.219.68:47475)
I20240501 20:09:06.350452 645 tablet_copy_service.cc:161] P c0931967d12847e08d9ff883c85f4237: Beginning new tablet copy session on tablet 09ac60ecdffa40b2a321c810d7890609 from peer 52541d349ee24edaac72825773fc848a at {username='slave'} at 127.29.219.68:47475: session id = 52541d349ee24edaac72825773fc848a-09ac60ecdffa40b2a321c810d7890609
I20240501 20:09:06.354259 645 tablet_copy_source_session.cc:205] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:06.360603 2597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09ac60ecdffa40b2a321c810d7890609. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.367511 2618 ts_tablet_manager.cc:902] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279: Initiating tablet copy from peer b811afcdb9d34a38b7e14e693b58093e (127.29.219.66:40067)
I20240501 20:09:06.373925 2597 tablet_copy_client.cc:783] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:06.374351 2597 tablet_copy_client.cc:647] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:06.377323 2597 tablet_copy_client.cc:523] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:06.377909 2623 ts_tablet_manager.cc:902] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0: Initiating tablet copy from peer c0931967d12847e08d9ff883c85f4237 (127.29.219.65:36897)
I20240501 20:09:06.381302 2597 tablet_bootstrap.cc:492] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a: Bootstrap starting.
I20240501 20:09:06.385035 2618 tablet_copy_client.cc:308] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.66:40067
I20240501 20:09:06.387035 2623 tablet_copy_client.cc:308] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.65:36897
I20240501 20:09:06.394822 2597 log.cc:824] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:06.397686 778 tablet_copy_service.cc:140] P b811afcdb9d34a38b7e14e693b58093e: Received BeginTabletCopySession request for tablet e606598eee444113b75b26943d884754 from peer 09cd35a57c9d4c0f86544e6ac1a55279 ({username='slave'} at 127.29.219.70:40983)
I20240501 20:09:06.398456 778 tablet_copy_service.cc:161] P b811afcdb9d34a38b7e14e693b58093e: Beginning new tablet copy session on tablet e606598eee444113b75b26943d884754 from peer 09cd35a57c9d4c0f86544e6ac1a55279 at {username='slave'} at 127.29.219.70:40983: session id = 09cd35a57c9d4c0f86544e6ac1a55279-e606598eee444113b75b26943d884754
I20240501 20:09:06.402549 778 tablet_copy_source_session.cc:205] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:06.408579 2618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e606598eee444113b75b26943d884754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.421861 2637 ts_tablet_manager.cc:902] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973: Initiating tablet copy from peer b811afcdb9d34a38b7e14e693b58093e (127.29.219.66:40067)
I20240501 20:09:06.427851 645 tablet_copy_service.cc:140] P c0931967d12847e08d9ff883c85f4237: Received BeginTabletCopySession request for tablet 2ad6ed0144eb4b619964ee4f86b9a842 from peer efa05351dbe344c7a451e7fdbeed47f0 ({username='slave'} at 127.29.219.69:48567)
I20240501 20:09:06.428094 645 tablet_copy_service.cc:161] P c0931967d12847e08d9ff883c85f4237: Beginning new tablet copy session on tablet 2ad6ed0144eb4b619964ee4f86b9a842 from peer efa05351dbe344c7a451e7fdbeed47f0 at {username='slave'} at 127.29.219.69:48567: session id = efa05351dbe344c7a451e7fdbeed47f0-2ad6ed0144eb4b619964ee4f86b9a842
I20240501 20:09:06.430797 2618 tablet_copy_client.cc:783] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:06.431128 645 tablet_copy_source_session.cc:205] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:06.431277 2618 tablet_copy_client.cc:647] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:06.434476 2618 tablet_copy_client.cc:523] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:06.436908 2623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ad6ed0144eb4b619964ee4f86b9a842. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.441653 2618 tablet_bootstrap.cc:492] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279: Bootstrap starting.
I20240501 20:09:06.453773 2637 tablet_copy_client.cc:308] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.66:40067
I20240501 20:09:06.459740 2618 log.cc:824] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:06.460974 2623 tablet_copy_client.cc:783] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:06.461371 2623 tablet_copy_client.cc:647] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:06.464890 2623 tablet_copy_client.cc:523] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:06.470728 2623 tablet_bootstrap.cc:492] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0: Bootstrap starting.
I20240501 20:09:06.476397 2597 tablet_bootstrap.cc:492] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:06.477216 2597 tablet_bootstrap.cc:492] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a: Bootstrap complete.
I20240501 20:09:06.477815 2597 ts_tablet_manager.cc:1367] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a: Time spent bootstrapping tablet: real 0.097s user 0.062s sys 0.033s
I20240501 20:09:06.483484 2648 ts_tablet_manager.cc:902] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973: Initiating tablet copy from peer c0931967d12847e08d9ff883c85f4237 (127.29.219.65:36897)
I20240501 20:09:06.485231 2597 raft_consensus.cc:348] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: true } }
I20240501 20:09:06.489588 778 tablet_copy_service.cc:140] P b811afcdb9d34a38b7e14e693b58093e: Received BeginTabletCopySession request for tablet 0a9d19f112154a65b012072f9d0d4149 from peer 52675e825f834cf3ac80a5777be65973 ({username='slave'} at 127.29.219.71:47407)
I20240501 20:09:06.489859 778 tablet_copy_service.cc:161] P b811afcdb9d34a38b7e14e693b58093e: Beginning new tablet copy session on tablet 0a9d19f112154a65b012072f9d0d4149 from peer 52675e825f834cf3ac80a5777be65973 at {username='slave'} at 127.29.219.71:47407: session id = 52675e825f834cf3ac80a5777be65973-0a9d19f112154a65b012072f9d0d4149
I20240501 20:09:06.494213 778 tablet_copy_source_session.cc:205] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:06.501466 2637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a9d19f112154a65b012072f9d0d4149. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.501756 2597 raft_consensus.cc:729] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 52541d349ee24edaac72825773fc848a, State: Initialized, Role: LEARNER
I20240501 20:09:06.502017 2623 log.cc:824] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:06.502552 2597 consensus_queue.cc:260] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: true } }
I20240501 20:09:06.516419 2597 ts_tablet_manager.cc:1398] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a: Time spent starting tablet: real 0.038s user 0.014s sys 0.007s
I20240501 20:09:06.517796 645 tablet_copy_service.cc:342] P c0931967d12847e08d9ff883c85f4237: Request end of tablet copy session 52541d349ee24edaac72825773fc848a-09ac60ecdffa40b2a321c810d7890609 received from {username='slave'} at 127.29.219.68:47475
I20240501 20:09:06.517968 645 tablet_copy_service.cc:433] P c0931967d12847e08d9ff883c85f4237: ending tablet copy session 52541d349ee24edaac72825773fc848a-09ac60ecdffa40b2a321c810d7890609 on tablet 09ac60ecdffa40b2a321c810d7890609 with peer 52541d349ee24edaac72825773fc848a
I20240501 20:09:06.518117 2660 ts_tablet_manager.cc:902] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a: Initiating tablet copy from peer b811afcdb9d34a38b7e14e693b58093e (127.29.219.66:40067)
I20240501 20:09:06.540630 2654 ts_tablet_manager.cc:902] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d: Initiating tablet copy from peer c0931967d12847e08d9ff883c85f4237 (127.29.219.65:36897)
I20240501 20:09:06.545922 2660 tablet_copy_client.cc:308] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.66:40067
I20240501 20:09:06.554363 778 tablet_copy_service.cc:140] P b811afcdb9d34a38b7e14e693b58093e: Received BeginTabletCopySession request for tablet 998f81b7cc7e4df7832f4694c4f34731 from peer 52541d349ee24edaac72825773fc848a ({username='slave'} at 127.29.219.68:58919)
I20240501 20:09:06.554648 778 tablet_copy_service.cc:161] P b811afcdb9d34a38b7e14e693b58093e: Beginning new tablet copy session on tablet 998f81b7cc7e4df7832f4694c4f34731 from peer 52541d349ee24edaac72825773fc848a at {username='slave'} at 127.29.219.68:58919: session id = 52541d349ee24edaac72825773fc848a-998f81b7cc7e4df7832f4694c4f34731
I20240501 20:09:06.558503 778 tablet_copy_source_session.cc:205] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:06.560436 2660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 998f81b7cc7e4df7832f4694c4f34731. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.575446 2660 tablet_copy_client.cc:783] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:06.575827 2660 tablet_copy_client.cc:647] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:06.578259 2660 tablet_copy_client.cc:523] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:06.583238 2675 ts_tablet_manager.cc:902] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d: Initiating tablet copy from peer c0931967d12847e08d9ff883c85f4237 (127.29.219.65:36897)
I20240501 20:09:06.584600 2637 tablet_copy_client.cc:783] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:06.584966 2674 ts_tablet_manager.cc:902] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0: Initiating tablet copy from peer b811afcdb9d34a38b7e14e693b58093e (127.29.219.66:40067)
I20240501 20:09:06.585028 2637 tablet_copy_client.cc:647] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:06.587538 2637 tablet_copy_client.cc:523] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:06.590226 2660 tablet_bootstrap.cc:492] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a: Bootstrap starting.
I20240501 20:09:06.592626 2648 tablet_copy_client.cc:308] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.65:36897
I20240501 20:09:06.602010 645 tablet_copy_service.cc:140] P c0931967d12847e08d9ff883c85f4237: Received BeginTabletCopySession request for tablet 5ae809988743421f8046716158b43a88 from peer 52675e825f834cf3ac80a5777be65973 ({username='slave'} at 127.29.219.71:48437)
I20240501 20:09:06.602226 645 tablet_copy_service.cc:161] P c0931967d12847e08d9ff883c85f4237: Beginning new tablet copy session on tablet 5ae809988743421f8046716158b43a88 from peer 52675e825f834cf3ac80a5777be65973 at {username='slave'} at 127.29.219.71:48437: session id = 52675e825f834cf3ac80a5777be65973-5ae809988743421f8046716158b43a88
I20240501 20:09:06.605273 645 tablet_copy_source_session.cc:205] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:06.606812 2648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae809988743421f8046716158b43a88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.616211 2648 tablet_copy_client.cc:783] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:06.616528 2648 tablet_copy_client.cc:647] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:06.617769 2618 tablet_bootstrap.cc:492] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:06.618384 2618 tablet_bootstrap.cc:492] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279: Bootstrap complete.
I20240501 20:09:06.618961 2618 ts_tablet_manager.cc:1367] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279: Time spent bootstrapping tablet: real 0.178s user 0.102s sys 0.024s
I20240501 20:09:06.618993 2648 tablet_copy_client.cc:523] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:06.620653 2637 tablet_bootstrap.cc:492] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973: Bootstrap starting.
I20240501 20:09:06.626080 2618 raft_consensus.cc:348] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: true } }
I20240501 20:09:06.626688 2618 raft_consensus.cc:729] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09cd35a57c9d4c0f86544e6ac1a55279, State: Initialized, Role: LEARNER
I20240501 20:09:06.627390 2618 consensus_queue.cc:260] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: true } }
I20240501 20:09:06.631112 2674 tablet_copy_client.cc:308] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.66:40067
I20240501 20:09:06.631747 2675 tablet_copy_client.cc:308] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.65:36897
I20240501 20:09:06.640267 2623 tablet_bootstrap.cc:492] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:06.641083 2623 tablet_bootstrap.cc:492] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0: Bootstrap complete.
I20240501 20:09:06.641762 2623 ts_tablet_manager.cc:1367] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0: Time spent bootstrapping tablet: real 0.171s user 0.073s sys 0.007s
I20240501 20:09:06.642529 778 tablet_copy_service.cc:140] P b811afcdb9d34a38b7e14e693b58093e: Received BeginTabletCopySession request for tablet 2705dcf15dc54cbbb263fcfee0ee8aed from peer efa05351dbe344c7a451e7fdbeed47f0 ({username='slave'} at 127.29.219.69:54345)
I20240501 20:09:06.642804 778 tablet_copy_service.cc:161] P b811afcdb9d34a38b7e14e693b58093e: Beginning new tablet copy session on tablet 2705dcf15dc54cbbb263fcfee0ee8aed from peer efa05351dbe344c7a451e7fdbeed47f0 at {username='slave'} at 127.29.219.69:54345: session id = efa05351dbe344c7a451e7fdbeed47f0-2705dcf15dc54cbbb263fcfee0ee8aed
I20240501 20:09:06.647202 778 tablet_copy_source_session.cc:205] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:06.649106 2674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2705dcf15dc54cbbb263fcfee0ee8aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.656214 2618 ts_tablet_manager.cc:1398] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279: Time spent starting tablet: real 0.037s user 0.016s sys 0.019s
I20240501 20:09:06.657212 778 tablet_copy_service.cc:342] P b811afcdb9d34a38b7e14e693b58093e: Request end of tablet copy session 09cd35a57c9d4c0f86544e6ac1a55279-e606598eee444113b75b26943d884754 received from {username='slave'} at 127.29.219.70:40983
I20240501 20:09:06.657430 778 tablet_copy_service.cc:433] P b811afcdb9d34a38b7e14e693b58093e: ending tablet copy session 09cd35a57c9d4c0f86544e6ac1a55279-e606598eee444113b75b26943d884754 on tablet e606598eee444113b75b26943d884754 with peer 09cd35a57c9d4c0f86544e6ac1a55279
I20240501 20:09:06.660145 2654 tablet_copy_client.cc:308] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.65:36897
I20240501 20:09:06.660575 2674 tablet_copy_client.cc:783] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:06.660921 2674 tablet_copy_client.cc:647] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:06.663022 2648 tablet_bootstrap.cc:492] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973: Bootstrap starting.
I20240501 20:09:06.663717 2674 tablet_copy_client.cc:523] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:06.667160 2674 tablet_bootstrap.cc:492] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0: Bootstrap starting.
I20240501 20:09:06.667923 2623 raft_consensus.cc:348] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: true } }
I20240501 20:09:06.668535 2623 raft_consensus.cc:729] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: efa05351dbe344c7a451e7fdbeed47f0, State: Initialized, Role: LEARNER
I20240501 20:09:06.669260 2623 consensus_queue.cc:260] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: true } }
I20240501 20:09:06.680478 2623 ts_tablet_manager.cc:1398] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0: Time spent starting tablet: real 0.039s user 0.008s sys 0.004s
I20240501 20:09:06.681924 645 tablet_copy_service.cc:342] P c0931967d12847e08d9ff883c85f4237: Request end of tablet copy session efa05351dbe344c7a451e7fdbeed47f0-2ad6ed0144eb4b619964ee4f86b9a842 received from {username='slave'} at 127.29.219.69:48567
I20240501 20:09:06.682089 645 tablet_copy_service.cc:433] P c0931967d12847e08d9ff883c85f4237: ending tablet copy session efa05351dbe344c7a451e7fdbeed47f0-2ad6ed0144eb4b619964ee4f86b9a842 on tablet 2ad6ed0144eb4b619964ee4f86b9a842 with peer efa05351dbe344c7a451e7fdbeed47f0
I20240501 20:09:06.687991 645 tablet_copy_service.cc:140] P c0931967d12847e08d9ff883c85f4237: Received BeginTabletCopySession request for tablet f71d46683ee74f0b8adb7ffdc4ec2688 from peer be64c56dce3d46fa80ba75efff719a7d ({username='slave'} at 127.29.219.67:44181)
I20240501 20:09:06.688009 644 tablet_copy_service.cc:140] P c0931967d12847e08d9ff883c85f4237: Received BeginTabletCopySession request for tablet 3a1f02449b91442a897ac2278ee98e29 from peer be64c56dce3d46fa80ba75efff719a7d ({username='slave'} at 127.29.219.67:44181)
I20240501 20:09:06.688248 645 tablet_copy_service.cc:161] P c0931967d12847e08d9ff883c85f4237: Beginning new tablet copy session on tablet f71d46683ee74f0b8adb7ffdc4ec2688 from peer be64c56dce3d46fa80ba75efff719a7d at {username='slave'} at 127.29.219.67:44181: session id = be64c56dce3d46fa80ba75efff719a7d-f71d46683ee74f0b8adb7ffdc4ec2688
I20240501 20:09:06.688536 644 tablet_copy_service.cc:161] P c0931967d12847e08d9ff883c85f4237: Beginning new tablet copy session on tablet 3a1f02449b91442a897ac2278ee98e29 from peer be64c56dce3d46fa80ba75efff719a7d at {username='slave'} at 127.29.219.67:44181: session id = be64c56dce3d46fa80ba75efff719a7d-3a1f02449b91442a897ac2278ee98e29
I20240501 20:09:06.691794 645 tablet_copy_source_session.cc:205] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:06.691797 644 tablet_copy_source_session.cc:205] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:06.693001 2660 tablet_bootstrap.cc:492] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:06.693569 2660 tablet_bootstrap.cc:492] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a: Bootstrap complete.
I20240501 20:09:06.693910 2660 ts_tablet_manager.cc:1367] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a: Time spent bootstrapping tablet: real 0.104s user 0.054s sys 0.006s
I20240501 20:09:06.694981 2660 raft_consensus.cc:348] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: true } }
I20240501 20:09:06.695277 2660 raft_consensus.cc:729] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 52541d349ee24edaac72825773fc848a, State: Initialized, Role: LEARNER
I20240501 20:09:06.695572 2660 consensus_queue.cc:260] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: true } }
I20240501 20:09:06.697398 2654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a1f02449b91442a897ac2278ee98e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.698799 2660 ts_tablet_manager.cc:1398] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:06.697170 2648 log.cc:824] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:06.701773 778 tablet_copy_service.cc:342] P b811afcdb9d34a38b7e14e693b58093e: Request end of tablet copy session 52541d349ee24edaac72825773fc848a-998f81b7cc7e4df7832f4694c4f34731 received from {username='slave'} at 127.29.219.68:58919
I20240501 20:09:06.701922 778 tablet_copy_service.cc:433] P b811afcdb9d34a38b7e14e693b58093e: ending tablet copy session 52541d349ee24edaac72825773fc848a-998f81b7cc7e4df7832f4694c4f34731 on tablet 998f81b7cc7e4df7832f4694c4f34731 with peer 52541d349ee24edaac72825773fc848a
I20240501 20:09:06.705466 2675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71d46683ee74f0b8adb7ffdc4ec2688. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:06.713943 2675 tablet_copy_client.cc:783] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:06.714246 2675 tablet_copy_client.cc:647] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:06.715355 2654 tablet_copy_client.cc:783] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:06.715591 2654 tablet_copy_client.cc:647] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:06.716820 2675 tablet_copy_client.cc:523] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:06.718925 2654 tablet_copy_client.cc:523] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:06.723865 2654 tablet_bootstrap.cc:492] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d: Bootstrap starting.
I20240501 20:09:06.728171 2674 tablet_bootstrap.cc:492] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:06.728621 2674 tablet_bootstrap.cc:492] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0: Bootstrap complete.
I20240501 20:09:06.728909 2674 ts_tablet_manager.cc:1367] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0: Time spent bootstrapping tablet: real 0.062s user 0.043s sys 0.014s
I20240501 20:09:06.729843 2674 raft_consensus.cc:348] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: true } }
I20240501 20:09:06.730098 2674 raft_consensus.cc:729] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: efa05351dbe344c7a451e7fdbeed47f0, State: Initialized, Role: LEARNER
I20240501 20:09:06.730306 2674 consensus_queue.cc:260] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: true } }
I20240501 20:09:06.730872 2674 ts_tablet_manager.cc:1398] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:06.731773 778 tablet_copy_service.cc:342] P b811afcdb9d34a38b7e14e693b58093e: Request end of tablet copy session efa05351dbe344c7a451e7fdbeed47f0-2705dcf15dc54cbbb263fcfee0ee8aed received from {username='slave'} at 127.29.219.69:54345
I20240501 20:09:06.731896 778 tablet_copy_service.cc:433] P b811afcdb9d34a38b7e14e693b58093e: ending tablet copy session efa05351dbe344c7a451e7fdbeed47f0-2705dcf15dc54cbbb263fcfee0ee8aed on tablet 2705dcf15dc54cbbb263fcfee0ee8aed with peer efa05351dbe344c7a451e7fdbeed47f0
I20240501 20:09:06.732301 2675 tablet_bootstrap.cc:492] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d: Bootstrap starting.
I20240501 20:09:06.743490 2675 log.cc:824] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:06.759047 2648 tablet_bootstrap.cc:492] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:06.759651 2648 tablet_bootstrap.cc:492] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973: Bootstrap complete.
I20240501 20:09:06.760139 2648 ts_tablet_manager.cc:1367] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973: Time spent bootstrapping tablet: real 0.097s user 0.065s sys 0.005s
I20240501 20:09:06.760437 2637 tablet_bootstrap.cc:492] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:06.761308 2637 tablet_bootstrap.cc:492] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973: Bootstrap complete.
I20240501 20:09:06.761556 2637 ts_tablet_manager.cc:1367] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973: Time spent bootstrapping tablet: real 0.143s user 0.055s sys 0.004s
I20240501 20:09:06.766687 2648 raft_consensus.cc:348] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: true } }
I20240501 20:09:06.766685 2637 raft_consensus.cc:348] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: true } }
I20240501 20:09:06.767182 2648 raft_consensus.cc:729] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 52675e825f834cf3ac80a5777be65973, State: Initialized, Role: LEARNER
I20240501 20:09:06.767366 2637 raft_consensus.cc:729] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 52675e825f834cf3ac80a5777be65973, State: Initialized, Role: LEARNER
I20240501 20:09:06.767789 2637 consensus_queue.cc:260] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: true } }
I20240501 20:09:06.767807 2648 consensus_queue.cc:260] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: true } }
I20240501 20:09:06.779426 2648 ts_tablet_manager.cc:1398] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973: Time spent starting tablet: real 0.019s user 0.020s sys 0.001s
I20240501 20:09:06.783085 2637 ts_tablet_manager.cc:1398] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973: Time spent starting tablet: real 0.021s user 0.005s sys 0.003s
I20240501 20:09:06.784391 778 tablet_copy_service.cc:342] P b811afcdb9d34a38b7e14e693b58093e: Request end of tablet copy session 52675e825f834cf3ac80a5777be65973-0a9d19f112154a65b012072f9d0d4149 received from {username='slave'} at 127.29.219.71:47407
I20240501 20:09:06.784540 778 tablet_copy_service.cc:433] P b811afcdb9d34a38b7e14e693b58093e: ending tablet copy session 52675e825f834cf3ac80a5777be65973-0a9d19f112154a65b012072f9d0d4149 on tablet 0a9d19f112154a65b012072f9d0d4149 with peer 52675e825f834cf3ac80a5777be65973
I20240501 20:09:06.785229 645 tablet_copy_service.cc:342] P c0931967d12847e08d9ff883c85f4237: Request end of tablet copy session 52675e825f834cf3ac80a5777be65973-5ae809988743421f8046716158b43a88 received from {username='slave'} at 127.29.219.71:48437
I20240501 20:09:06.785358 645 tablet_copy_service.cc:433] P c0931967d12847e08d9ff883c85f4237: ending tablet copy session 52675e825f834cf3ac80a5777be65973-5ae809988743421f8046716158b43a88 on tablet 5ae809988743421f8046716158b43a88 with peer 52675e825f834cf3ac80a5777be65973
I20240501 20:09:06.805203 2675 tablet_bootstrap.cc:492] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:06.805806 2675 tablet_bootstrap.cc:492] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d: Bootstrap complete.
I20240501 20:09:06.806309 2654 tablet_bootstrap.cc:492] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:06.806316 2675 ts_tablet_manager.cc:1367] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d: Time spent bootstrapping tablet: real 0.074s user 0.050s sys 0.013s
I20240501 20:09:06.806799 2654 tablet_bootstrap.cc:492] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d: Bootstrap complete.
I20240501 20:09:06.807060 2654 ts_tablet_manager.cc:1367] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d: Time spent bootstrapping tablet: real 0.083s user 0.054s sys 0.018s
I20240501 20:09:06.811874 2675 raft_consensus.cc:348] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: true } }
I20240501 20:09:06.811761 2654 raft_consensus.cc:348] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: true } }
I20240501 20:09:06.812304 2675 raft_consensus.cc:729] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be64c56dce3d46fa80ba75efff719a7d, State: Initialized, Role: LEARNER
I20240501 20:09:06.812312 2654 raft_consensus.cc:729] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be64c56dce3d46fa80ba75efff719a7d, State: Initialized, Role: LEARNER
I20240501 20:09:06.812945 2654 consensus_queue.cc:260] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: true } }
I20240501 20:09:06.812952 2675 consensus_queue.cc:260] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: true } }
I20240501 20:09:06.815482 2654 ts_tablet_manager.cc:1398] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d: Time spent starting tablet: real 0.008s user 0.004s sys 0.001s
I20240501 20:09:06.815543 2675 ts_tablet_manager.cc:1398] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20240501 20:09:06.817803 645 tablet_copy_service.cc:342] P c0931967d12847e08d9ff883c85f4237: Request end of tablet copy session be64c56dce3d46fa80ba75efff719a7d-f71d46683ee74f0b8adb7ffdc4ec2688 received from {username='slave'} at 127.29.219.67:44181
I20240501 20:09:06.817936 645 tablet_copy_service.cc:433] P c0931967d12847e08d9ff883c85f4237: ending tablet copy session be64c56dce3d46fa80ba75efff719a7d-f71d46683ee74f0b8adb7ffdc4ec2688 on tablet f71d46683ee74f0b8adb7ffdc4ec2688 with peer be64c56dce3d46fa80ba75efff719a7d
I20240501 20:09:06.818249 644 tablet_copy_service.cc:342] P c0931967d12847e08d9ff883c85f4237: Request end of tablet copy session be64c56dce3d46fa80ba75efff719a7d-3a1f02449b91442a897ac2278ee98e29 received from {username='slave'} at 127.29.219.67:44181
I20240501 20:09:06.818401 644 tablet_copy_service.cc:433] P c0931967d12847e08d9ff883c85f4237: ending tablet copy session be64c56dce3d46fa80ba75efff719a7d-3a1f02449b91442a897ac2278ee98e29 on tablet 3a1f02449b91442a897ac2278ee98e29 with peer be64c56dce3d46fa80ba75efff719a7d
I20240501 20:09:06.828433 1954 raft_consensus.cc:1203] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:09:06.852830 2215 raft_consensus.cc:1203] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:09:06.881520 2084 raft_consensus.cc:1203] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:09:06.934693 2346 raft_consensus.cc:1203] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:09:06.958891 2346 raft_consensus.cc:1203] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:09:06.999974 1822 raft_consensus.cc:1203] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:09:07.041895 1822 raft_consensus.cc:1203] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:09:07.063635 2084 raft_consensus.cc:1203] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:09:07.083711 1954 raft_consensus.cc:1203] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:09:07.223022 2494 raft_consensus.cc:1050] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237: attempting to promote NON_VOTER 52541d349ee24edaac72825773fc848a to VOTER
I20240501 20:09:07.223842 2494 consensus_queue.cc:237] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: false } }
I20240501 20:09:07.225709 1954 raft_consensus.cc:1261] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a [term 1 LEARNER]: Refusing update from remote peer c0931967d12847e08d9ff883c85f4237: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:09:07.226380 2705 consensus_queue.cc:1035] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:07.229645 2494 raft_consensus.cc:2932] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 52541d349ee24edaac72825773fc848a (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: false } } }
I20240501 20:09:07.230330 1954 raft_consensus.cc:2932] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 52541d349ee24edaac72825773fc848a (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: false } } }
I20240501 20:09:07.233872 475 catalog_manager.cc:5484] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 reported cstate change: config changed from index 4 to 5, 52541d349ee24edaac72825773fc848a (127.29.219.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c0931967d12847e08d9ff883c85f4237" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:07.254348 2718 raft_consensus.cc:1050] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e: attempting to promote NON_VOTER 09cd35a57c9d4c0f86544e6ac1a55279 to VOTER
I20240501 20:09:07.255240 2718 consensus_queue.cc:237] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: false } }
I20240501 20:09:07.257228 2215 raft_consensus.cc:1261] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279 [term 1 LEARNER]: Refusing update from remote peer b811afcdb9d34a38b7e14e693b58093e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:09:07.257894 2481 consensus_queue.cc:1035] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [LEADER]: Connected to new peer: Peer: permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:07.261893 2718 raft_consensus.cc:2932] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 09cd35a57c9d4c0f86544e6ac1a55279 (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: false } } }
I20240501 20:09:07.262710 2215 raft_consensus.cc:2932] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 09cd35a57c9d4c0f86544e6ac1a55279 (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: false } } }
I20240501 20:09:07.267297 475 catalog_manager.cc:5484] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279 reported cstate change: config changed from index 4 to 5, 09cd35a57c9d4c0f86544e6ac1a55279 (127.29.219.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b811afcdb9d34a38b7e14e693b58093e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: false } } }
I20240501 20:09:07.336052 2705 raft_consensus.cc:1050] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237: attempting to promote NON_VOTER efa05351dbe344c7a451e7fdbeed47f0 to VOTER
I20240501 20:09:07.336712 2705 consensus_queue.cc:237] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: false } }
I20240501 20:09:07.338793 2084 raft_consensus.cc:1261] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0 [term 1 LEARNER]: Refusing update from remote peer c0931967d12847e08d9ff883c85f4237: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:09:07.339344 2494 consensus_queue.cc:1035] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:07.342661 2705 raft_consensus.cc:2932] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efa05351dbe344c7a451e7fdbeed47f0 (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: false } } }
I20240501 20:09:07.343323 2084 raft_consensus.cc:2932] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efa05351dbe344c7a451e7fdbeed47f0 (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: false } } }
I20240501 20:09:07.347036 482 catalog_manager.cc:5484] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 reported cstate change: config changed from index 4 to 5, efa05351dbe344c7a451e7fdbeed47f0 (127.29.219.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c0931967d12847e08d9ff883c85f4237" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:07.387558 2706 raft_consensus.cc:1050] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237: attempting to promote NON_VOTER 52675e825f834cf3ac80a5777be65973 to VOTER
I20240501 20:09:07.388161 2706 consensus_queue.cc:237] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: false } }
I20240501 20:09:07.390383 2346 raft_consensus.cc:1261] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973 [term 1 LEARNER]: Refusing update from remote peer c0931967d12847e08d9ff883c85f4237: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:09:07.391064 2494 consensus_queue.cc:1035] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:07.394804 2706 raft_consensus.cc:2932] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 52675e825f834cf3ac80a5777be65973 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: false } } }
I20240501 20:09:07.395697 2346 raft_consensus.cc:2932] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 52675e825f834cf3ac80a5777be65973 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: false } } }
I20240501 20:09:07.400465 2718 raft_consensus.cc:1050] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e: attempting to promote NON_VOTER 52675e825f834cf3ac80a5777be65973 to VOTER
I20240501 20:09:07.401094 2718 consensus_queue.cc:237] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: false } }
I20240501 20:09:07.404122 475 catalog_manager.cc:5484] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 reported cstate change: config changed from index 4 to 5, 52675e825f834cf3ac80a5777be65973 (127.29.219.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c0931967d12847e08d9ff883c85f4237" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:07.410197 2706 raft_consensus.cc:1050] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237: attempting to promote NON_VOTER be64c56dce3d46fa80ba75efff719a7d to VOTER
I20240501 20:09:07.411151 2706 consensus_queue.cc:237] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: false } }
I20240501 20:09:07.414876 2345 raft_consensus.cc:1261] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973 [term 1 LEARNER]: Refusing update from remote peer b811afcdb9d34a38b7e14e693b58093e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:09:07.415616 2481 consensus_queue.cc:1035] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [LEADER]: Connected to new peer: Peer: permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:07.420970 2717 raft_consensus.cc:2932] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 52675e825f834cf3ac80a5777be65973 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: false } } }
I20240501 20:09:07.421768 2345 raft_consensus.cc:2932] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 52675e825f834cf3ac80a5777be65973 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: false } } }
I20240501 20:09:07.425686 1822 raft_consensus.cc:1261] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d [term 1 LEARNER]: Refusing update from remote peer c0931967d12847e08d9ff883c85f4237: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:09:07.426512 2705 consensus_queue.cc:1035] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:07.427927 475 catalog_manager.cc:5484] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973 reported cstate change: config changed from index 4 to 5, 52675e825f834cf3ac80a5777be65973 (127.29.219.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b811afcdb9d34a38b7e14e693b58093e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: false } } }
I20240501 20:09:07.431596 2494 raft_consensus.cc:2932] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be64c56dce3d46fa80ba75efff719a7d (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: false } } }
I20240501 20:09:07.432314 1822 raft_consensus.cc:2932] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be64c56dce3d46fa80ba75efff719a7d (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: false } } }
I20240501 20:09:07.436383 482 catalog_manager.cc:5484] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 reported cstate change: config changed from index 4 to 5, be64c56dce3d46fa80ba75efff719a7d (127.29.219.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c0931967d12847e08d9ff883c85f4237" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:07.457916 625 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f71d46683ee74f0b8adb7ffdc4ec2688"
dest_uuid: "c0931967d12847e08d9ff883c85f4237"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41316
I20240501 20:09:07.458212 625 raft_consensus.cc:595] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:07.485173 758 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0a9d19f112154a65b012072f9d0d4149"
dest_uuid: "b811afcdb9d34a38b7e14e693b58093e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37912
I20240501 20:09:07.485433 758 raft_consensus.cc:595] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:07.503928 625 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2ad6ed0144eb4b619964ee4f86b9a842"
dest_uuid: "c0931967d12847e08d9ff883c85f4237"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41332
I20240501 20:09:07.504139 625 raft_consensus.cc:595] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:07.530207 625 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "09ac60ecdffa40b2a321c810d7890609"
dest_uuid: "c0931967d12847e08d9ff883c85f4237"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41350
I20240501 20:09:07.530445 625 raft_consensus.cc:595] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:07.548542 758 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e606598eee444113b75b26943d884754"
dest_uuid: "b811afcdb9d34a38b7e14e693b58093e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37936
I20240501 20:09:07.548765 758 raft_consensus.cc:595] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:07.566401 625 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5ae809988743421f8046716158b43a88"
dest_uuid: "c0931967d12847e08d9ff883c85f4237"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41364
I20240501 20:09:07.566720 625 raft_consensus.cc:595] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:07.602406 2718 raft_consensus.cc:1050] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e: attempting to promote NON_VOTER efa05351dbe344c7a451e7fdbeed47f0 to VOTER
I20240501 20:09:07.603066 2718 consensus_queue.cc:237] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: false } }
I20240501 20:09:07.604828 2084 raft_consensus.cc:1261] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0 [term 1 LEARNER]: Refusing update from remote peer b811afcdb9d34a38b7e14e693b58093e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:09:07.605336 2481 consensus_queue.cc:1035] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [LEADER]: Connected to new peer: Peer: permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:07.607709 2718 raft_consensus.cc:2932] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efa05351dbe344c7a451e7fdbeed47f0 (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: false } } }
I20240501 20:09:07.608328 2084 raft_consensus.cc:2932] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, efa05351dbe344c7a451e7fdbeed47f0 (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: false } } }
I20240501 20:09:07.611759 482 catalog_manager.cc:5484] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e reported cstate change: config changed from index 4 to 5, efa05351dbe344c7a451e7fdbeed47f0 (127.29.219.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b811afcdb9d34a38b7e14e693b58093e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:07.641054 2705 raft_consensus.cc:1050] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237: attempting to promote NON_VOTER be64c56dce3d46fa80ba75efff719a7d to VOTER
I20240501 20:09:07.641793 2705 consensus_queue.cc:237] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: false } }
I20240501 20:09:07.643545 1822 raft_consensus.cc:1261] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d [term 1 LEARNER]: Refusing update from remote peer c0931967d12847e08d9ff883c85f4237: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:09:07.644055 2494 consensus_queue.cc:1035] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:07.646478 2705 raft_consensus.cc:2932] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be64c56dce3d46fa80ba75efff719a7d (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: false } } }
I20240501 20:09:07.647224 1822 raft_consensus.cc:2932] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, be64c56dce3d46fa80ba75efff719a7d (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: false } } }
I20240501 20:09:07.651753 483 catalog_manager.cc:5484] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 reported cstate change: config changed from index 4 to 5, be64c56dce3d46fa80ba75efff719a7d (127.29.219.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c0931967d12847e08d9ff883c85f4237" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:07.657261 2718 raft_consensus.cc:1050] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e: attempting to promote NON_VOTER 52541d349ee24edaac72825773fc848a to VOTER
I20240501 20:09:07.657719 2718 consensus_queue.cc:237] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: false } }
I20240501 20:09:07.659350 1954 raft_consensus.cc:1261] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a [term 1 LEARNER]: Refusing update from remote peer b811afcdb9d34a38b7e14e693b58093e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:09:07.659848 2718 consensus_queue.cc:1035] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [LEADER]: Connected to new peer: Peer: permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:07.662202 2717 raft_consensus.cc:2932] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 52541d349ee24edaac72825773fc848a (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: false } } }
I20240501 20:09:07.662863 1954 raft_consensus.cc:2932] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 52541d349ee24edaac72825773fc848a (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: false } } }
I20240501 20:09:07.667266 482 catalog_manager.cc:5484] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e reported cstate change: config changed from index 4 to 5, 52541d349ee24edaac72825773fc848a (127.29.219.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b811afcdb9d34a38b7e14e693b58093e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:07.765718 2717 raft_consensus.cc:979] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e: : Instructing follower 09cd35a57c9d4c0f86544e6ac1a55279 to start an election
I20240501 20:09:07.765892 2717 raft_consensus.cc:1067] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Signalling peer 09cd35a57c9d4c0f86544e6ac1a55279 to start an election
I20240501 20:09:07.766813 2215 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e606598eee444113b75b26943d884754"
dest_uuid: "09cd35a57c9d4c0f86544e6ac1a55279"
from {username='slave'} at 127.29.219.66:51291
I20240501 20:09:07.767093 2215 raft_consensus.cc:482] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:07.767307 2215 raft_consensus.cc:3037] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:07.770033 2215 raft_consensus.cc:504] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: false } }
I20240501 20:09:07.771392 2215 leader_election.cc:288] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279 [CANDIDATE]: Term 2 election: Requested vote from peers b811afcdb9d34a38b7e14e693b58093e (127.29.219.66:40067)
I20240501 20:09:07.772253 758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e606598eee444113b75b26943d884754" candidate_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b811afcdb9d34a38b7e14e693b58093e"
I20240501 20:09:07.772563 758 raft_consensus.cc:3032] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:07.772675 758 raft_consensus.cc:729] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b811afcdb9d34a38b7e14e693b58093e, State: Running, Role: LEADER
I20240501 20:09:07.772959 758 consensus_queue.cc:260] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: false } }
I20240501 20:09:07.773276 758 raft_consensus.cc:3037] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:07.775314 758 raft_consensus.cc:2454] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09cd35a57c9d4c0f86544e6ac1a55279 in term 2.
I20240501 20:09:07.775846 2147 leader_election.cc:302] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 09cd35a57c9d4c0f86544e6ac1a55279, b811afcdb9d34a38b7e14e693b58093e; no voters:
I20240501 20:09:07.776468 2899 raft_consensus.cc:2781] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:07.776940 2899 raft_consensus.cc:686] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279 [term 2 LEADER]: Becoming Leader. State: Replica: 09cd35a57c9d4c0f86544e6ac1a55279, State: Running, Role: LEADER
I20240501 20:09:07.777465 2899 consensus_queue.cc:237] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: false } }
I20240501 20:09:07.781961 483 catalog_manager.cc:5484] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279 reported cstate change: term changed from 1 to 2, leader changed from b811afcdb9d34a38b7e14e693b58093e (127.29.219.66) to 09cd35a57c9d4c0f86544e6ac1a55279 (127.29.219.70). New cstate: current_term: 2 leader_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:09:07.794281 470 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet e606598eee444113b75b26943d884754 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 09cd35a57c9d4c0f86544e6ac1a55279 (127.29.219.70:40955) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240501 20:09:07.801903 2494 raft_consensus.cc:979] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237: : Instructing follower 52541d349ee24edaac72825773fc848a to start an election
I20240501 20:09:07.802129 2494 raft_consensus.cc:1067] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Signalling peer 52541d349ee24edaac72825773fc848a to start an election
I20240501 20:09:07.804126 1954 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "09ac60ecdffa40b2a321c810d7890609"
dest_uuid: "52541d349ee24edaac72825773fc848a"
from {username='slave'} at 127.29.219.65:54923
I20240501 20:09:07.804414 1954 raft_consensus.cc:482] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:07.804574 1954 raft_consensus.cc:3037] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:07.807509 1954 raft_consensus.cc:504] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: false } }
I20240501 20:09:07.807832 758 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "998f81b7cc7e4df7832f4694c4f34731"
dest_uuid: "b811afcdb9d34a38b7e14e693b58093e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37956
I20240501 20:09:07.808018 758 raft_consensus.cc:595] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:07.809867 625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09ac60ecdffa40b2a321c810d7890609" candidate_uuid: "52541d349ee24edaac72825773fc848a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c0931967d12847e08d9ff883c85f4237"
I20240501 20:09:07.810179 625 raft_consensus.cc:3032] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:07.809701 1954 leader_election.cc:288] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a [CANDIDATE]: Term 2 election: Requested vote from peers c0931967d12847e08d9ff883c85f4237 (127.29.219.65:36897)
I20240501 20:09:07.810285 625 raft_consensus.cc:729] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0931967d12847e08d9ff883c85f4237, State: Running, Role: LEADER
I20240501 20:09:07.810616 625 consensus_queue.cc:260] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: false } }
I20240501 20:09:07.810995 625 raft_consensus.cc:3037] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:07.814352 625 raft_consensus.cc:2454] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52541d349ee24edaac72825773fc848a in term 2.
I20240501 20:09:07.815044 1888 leader_election.cc:302] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 52541d349ee24edaac72825773fc848a, c0931967d12847e08d9ff883c85f4237; no voters:
I20240501 20:09:07.815420 2653 raft_consensus.cc:2781] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:07.815910 2653 raft_consensus.cc:686] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a [term 2 LEADER]: Becoming Leader. State: Replica: 52541d349ee24edaac72825773fc848a, State: Running, Role: LEADER
I20240501 20:09:07.816407 2653 consensus_queue.cc:237] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: false } }
I20240501 20:09:07.820029 483 catalog_manager.cc:5484] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a reported cstate change: term changed from 1 to 2, leader changed from c0931967d12847e08d9ff883c85f4237 (127.29.219.65) to 52541d349ee24edaac72825773fc848a (127.29.219.68). New cstate: current_term: 2 leader_uuid: "52541d349ee24edaac72825773fc848a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:07.830582 625 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f71d46683ee74f0b8adb7ffdc4ec2688"
dest_uuid: "c0931967d12847e08d9ff883c85f4237"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41382
I20240501 20:09:07.830899 625 raft_consensus.cc:595] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:07.848667 758 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2705dcf15dc54cbbb263fcfee0ee8aed"
dest_uuid: "b811afcdb9d34a38b7e14e693b58093e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37986
I20240501 20:09:07.848912 758 raft_consensus.cc:595] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:07.865074 2705 raft_consensus.cc:979] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237: : Instructing follower efa05351dbe344c7a451e7fdbeed47f0 to start an election
I20240501 20:09:07.865249 2705 raft_consensus.cc:1067] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Signalling peer efa05351dbe344c7a451e7fdbeed47f0 to start an election
I20240501 20:09:07.865965 2084 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2ad6ed0144eb4b619964ee4f86b9a842"
dest_uuid: "efa05351dbe344c7a451e7fdbeed47f0"
from {username='slave'} at 127.29.219.65:48249
I20240501 20:09:07.866308 2084 raft_consensus.cc:482] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:07.866498 2084 raft_consensus.cc:3037] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:07.868811 2084 raft_consensus.cc:504] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: false } }
I20240501 20:09:07.870015 2084 leader_election.cc:288] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0 [CANDIDATE]: Term 2 election: Requested vote from peers c0931967d12847e08d9ff883c85f4237 (127.29.219.65:36897)
I20240501 20:09:07.870513 625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ad6ed0144eb4b619964ee4f86b9a842" candidate_uuid: "efa05351dbe344c7a451e7fdbeed47f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c0931967d12847e08d9ff883c85f4237"
I20240501 20:09:07.870796 625 raft_consensus.cc:3032] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:07.870793 758 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0a9d19f112154a65b012072f9d0d4149"
dest_uuid: "b811afcdb9d34a38b7e14e693b58093e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38000
I20240501 20:09:07.870883 625 raft_consensus.cc:729] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0931967d12847e08d9ff883c85f4237, State: Running, Role: LEADER
I20240501 20:09:07.870965 758 raft_consensus.cc:595] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:07.871174 625 consensus_queue.cc:260] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: false } }
I20240501 20:09:07.871475 625 raft_consensus.cc:3037] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:07.873790 625 raft_consensus.cc:2454] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efa05351dbe344c7a451e7fdbeed47f0 in term 2.
I20240501 20:09:07.874312 2019 leader_election.cc:302] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: c0931967d12847e08d9ff883c85f4237, efa05351dbe344c7a451e7fdbeed47f0; no voters:
I20240501 20:09:07.874754 2690 raft_consensus.cc:2781] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:07.875631 2690 raft_consensus.cc:686] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0 [term 2 LEADER]: Becoming Leader. State: Replica: efa05351dbe344c7a451e7fdbeed47f0, State: Running, Role: LEADER
I20240501 20:09:07.876123 2690 consensus_queue.cc:237] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: false } }
I20240501 20:09:07.879848 483 catalog_manager.cc:5484] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0 reported cstate change: term changed from 1 to 2, leader changed from c0931967d12847e08d9ff883c85f4237 (127.29.219.65) to efa05351dbe344c7a451e7fdbeed47f0 (127.29.219.69). New cstate: current_term: 2 leader_uuid: "efa05351dbe344c7a451e7fdbeed47f0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:07.902374 2494 raft_consensus.cc:979] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237: : Instructing follower 52675e825f834cf3ac80a5777be65973 to start an election
I20240501 20:09:07.902606 2494 raft_consensus.cc:1067] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Signalling peer 52675e825f834cf3ac80a5777be65973 to start an election
I20240501 20:09:07.903795 2345 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5ae809988743421f8046716158b43a88"
dest_uuid: "52675e825f834cf3ac80a5777be65973"
from {username='slave'} at 127.29.219.65:55809
I20240501 20:09:07.904098 2345 raft_consensus.cc:482] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:07.904316 2345 raft_consensus.cc:3037] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:07.905853 625 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3a1f02449b91442a897ac2278ee98e29"
dest_uuid: "c0931967d12847e08d9ff883c85f4237"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41396
I20240501 20:09:07.906075 625 raft_consensus.cc:595] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:07.907603 2345 raft_consensus.cc:504] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: false } }
I20240501 20:09:07.909101 2345 leader_election.cc:288] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973 [CANDIDATE]: Term 2 election: Requested vote from peers c0931967d12847e08d9ff883c85f4237 (127.29.219.65:36897)
I20240501 20:09:07.909678 625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae809988743421f8046716158b43a88" candidate_uuid: "52675e825f834cf3ac80a5777be65973" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c0931967d12847e08d9ff883c85f4237"
I20240501 20:09:07.909909 625 raft_consensus.cc:3032] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:07.909996 625 raft_consensus.cc:729] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0931967d12847e08d9ff883c85f4237, State: Running, Role: LEADER
I20240501 20:09:07.910320 625 consensus_queue.cc:260] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: false } }
I20240501 20:09:07.910697 625 raft_consensus.cc:3037] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:07.912840 625 raft_consensus.cc:2454] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52675e825f834cf3ac80a5777be65973 in term 2.
I20240501 20:09:07.913326 2280 leader_election.cc:302] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 52675e825f834cf3ac80a5777be65973, c0931967d12847e08d9ff883c85f4237; no voters:
I20240501 20:09:07.913750 2697 raft_consensus.cc:2781] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:07.914167 2697 raft_consensus.cc:686] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973 [term 2 LEADER]: Becoming Leader. State: Replica: 52675e825f834cf3ac80a5777be65973, State: Running, Role: LEADER
I20240501 20:09:07.914644 2697 consensus_queue.cc:237] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: false } }
I20240501 20:09:07.918152 483 catalog_manager.cc:5484] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973 reported cstate change: term changed from 1 to 2, leader changed from c0931967d12847e08d9ff883c85f4237 (127.29.219.65) to 52675e825f834cf3ac80a5777be65973 (127.29.219.71). New cstate: current_term: 2 leader_uuid: "52675e825f834cf3ac80a5777be65973" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:07.991134 2705 raft_consensus.cc:979] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237: : Instructing follower be64c56dce3d46fa80ba75efff719a7d to start an election
I20240501 20:09:07.991315 2705 raft_consensus.cc:1067] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Signalling peer be64c56dce3d46fa80ba75efff719a7d to start an election
I20240501 20:09:07.992077 1822 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f71d46683ee74f0b8adb7ffdc4ec2688"
dest_uuid: "be64c56dce3d46fa80ba75efff719a7d"
from {username='slave'} at 127.29.219.65:43049
I20240501 20:09:07.992345 1822 raft_consensus.cc:482] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:07.992527 1822 raft_consensus.cc:3037] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:07.994884 1822 raft_consensus.cc:504] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: false } }
I20240501 20:09:07.996181 1822 leader_election.cc:288] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d [CANDIDATE]: Term 2 election: Requested vote from peers c0931967d12847e08d9ff883c85f4237 (127.29.219.65:36897)
I20240501 20:09:07.996701 625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71d46683ee74f0b8adb7ffdc4ec2688" candidate_uuid: "be64c56dce3d46fa80ba75efff719a7d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c0931967d12847e08d9ff883c85f4237"
I20240501 20:09:07.996909 625 raft_consensus.cc:3032] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:07.996991 625 raft_consensus.cc:729] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0931967d12847e08d9ff883c85f4237, State: Running, Role: LEADER
I20240501 20:09:07.997246 625 consensus_queue.cc:260] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: false } }
I20240501 20:09:07.997491 625 raft_consensus.cc:3037] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:07.999668 625 raft_consensus.cc:2454] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be64c56dce3d46fa80ba75efff719a7d in term 2.
I20240501 20:09:08.000162 1756 leader_election.cc:302] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: be64c56dce3d46fa80ba75efff719a7d, c0931967d12847e08d9ff883c85f4237; no voters:
I20240501 20:09:08.000547 2702 raft_consensus.cc:2781] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:08.000949 2702 raft_consensus.cc:686] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d [term 2 LEADER]: Becoming Leader. State: Replica: be64c56dce3d46fa80ba75efff719a7d, State: Running, Role: LEADER
I20240501 20:09:08.001369 2702 consensus_queue.cc:237] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: false } }
I20240501 20:09:08.004420 483 catalog_manager.cc:5484] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d reported cstate change: term changed from 1 to 2, leader changed from c0931967d12847e08d9ff883c85f4237 (127.29.219.65) to be64c56dce3d46fa80ba75efff719a7d (127.29.219.67). New cstate: current_term: 2 leader_uuid: "be64c56dce3d46fa80ba75efff719a7d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:08.029000 2481 raft_consensus.cc:979] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e: : Instructing follower 52675e825f834cf3ac80a5777be65973 to start an election
I20240501 20:09:08.029206 2481 raft_consensus.cc:1067] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Signalling peer 52675e825f834cf3ac80a5777be65973 to start an election
I20240501 20:09:08.029855 2345 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0a9d19f112154a65b012072f9d0d4149"
dest_uuid: "52675e825f834cf3ac80a5777be65973"
from {username='slave'} at 127.29.219.66:42419
I20240501 20:09:08.030087 2345 raft_consensus.cc:482] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:08.030187 2345 raft_consensus.cc:3037] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:08.032981 2345 raft_consensus.cc:504] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: false } }
I20240501 20:09:08.033623 2345 leader_election.cc:288] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973 [CANDIDATE]: Term 2 election: Requested vote from peers b811afcdb9d34a38b7e14e693b58093e (127.29.219.66:40067)
I20240501 20:09:08.034207 758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a9d19f112154a65b012072f9d0d4149" candidate_uuid: "52675e825f834cf3ac80a5777be65973" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b811afcdb9d34a38b7e14e693b58093e"
I20240501 20:09:08.034404 758 raft_consensus.cc:3032] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:08.034502 758 raft_consensus.cc:729] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b811afcdb9d34a38b7e14e693b58093e, State: Running, Role: LEADER
I20240501 20:09:08.034778 758 consensus_queue.cc:260] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: false } }
I20240501 20:09:08.035038 758 raft_consensus.cc:3037] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:08.037026 758 raft_consensus.cc:2454] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52675e825f834cf3ac80a5777be65973 in term 2.
I20240501 20:09:08.037473 2278 leader_election.cc:302] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 52675e825f834cf3ac80a5777be65973, b811afcdb9d34a38b7e14e693b58093e; no voters:
I20240501 20:09:08.037748 2697 raft_consensus.cc:2781] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:08.037901 2697 raft_consensus.cc:686] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973 [term 2 LEADER]: Becoming Leader. State: Replica: 52675e825f834cf3ac80a5777be65973, State: Running, Role: LEADER
I20240501 20:09:08.038152 2697 consensus_queue.cc:237] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: false } }
I20240501 20:09:08.040161 483 catalog_manager.cc:5484] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973 reported cstate change: term changed from 1 to 2, leader changed from b811afcdb9d34a38b7e14e693b58093e (127.29.219.66) to 52675e825f834cf3ac80a5777be65973 (127.29.219.71). New cstate: current_term: 2 leader_uuid: "52675e825f834cf3ac80a5777be65973" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:08.058426 2717 raft_consensus.cc:979] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e: : Instructing follower efa05351dbe344c7a451e7fdbeed47f0 to start an election
I20240501 20:09:08.058591 2717 raft_consensus.cc:1067] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Signalling peer efa05351dbe344c7a451e7fdbeed47f0 to start an election
I20240501 20:09:08.059254 2084 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2705dcf15dc54cbbb263fcfee0ee8aed"
dest_uuid: "efa05351dbe344c7a451e7fdbeed47f0"
from {username='slave'} at 127.29.219.66:46359
I20240501 20:09:08.059458 2084 raft_consensus.cc:482] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:08.059563 2084 raft_consensus.cc:3037] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:08.062362 2084 raft_consensus.cc:504] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: false } }
I20240501 20:09:08.063114 2084 leader_election.cc:288] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0 [CANDIDATE]: Term 2 election: Requested vote from peers b811afcdb9d34a38b7e14e693b58093e (127.29.219.66:40067)
I20240501 20:09:08.063894 758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2705dcf15dc54cbbb263fcfee0ee8aed" candidate_uuid: "efa05351dbe344c7a451e7fdbeed47f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b811afcdb9d34a38b7e14e693b58093e"
I20240501 20:09:08.064102 758 raft_consensus.cc:3032] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:08.064208 758 raft_consensus.cc:729] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b811afcdb9d34a38b7e14e693b58093e, State: Running, Role: LEADER
I20240501 20:09:08.064466 758 consensus_queue.cc:260] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: false } }
I20240501 20:09:08.064796 758 raft_consensus.cc:3037] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:08.066800 758 raft_consensus.cc:2454] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efa05351dbe344c7a451e7fdbeed47f0 in term 2.
I20240501 20:09:08.067149 2017 leader_election.cc:302] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b811afcdb9d34a38b7e14e693b58093e, efa05351dbe344c7a451e7fdbeed47f0; no voters:
I20240501 20:09:08.067433 2690 raft_consensus.cc:2781] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:08.067617 2690 raft_consensus.cc:686] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0 [term 2 LEADER]: Becoming Leader. State: Replica: efa05351dbe344c7a451e7fdbeed47f0, State: Running, Role: LEADER
I20240501 20:09:08.067855 2690 consensus_queue.cc:237] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: false } }
I20240501 20:09:08.069882 483 catalog_manager.cc:5484] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0 reported cstate change: term changed from 1 to 2, leader changed from b811afcdb9d34a38b7e14e693b58093e (127.29.219.66) to efa05351dbe344c7a451e7fdbeed47f0 (127.29.219.69). New cstate: current_term: 2 leader_uuid: "efa05351dbe344c7a451e7fdbeed47f0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:08.149722 2494 raft_consensus.cc:979] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237: : Instructing follower be64c56dce3d46fa80ba75efff719a7d to start an election
I20240501 20:09:08.149897 2494 raft_consensus.cc:1067] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Signalling peer be64c56dce3d46fa80ba75efff719a7d to start an election
I20240501 20:09:08.150547 1822 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3a1f02449b91442a897ac2278ee98e29"
dest_uuid: "be64c56dce3d46fa80ba75efff719a7d"
from {username='slave'} at 127.29.219.65:43049
I20240501 20:09:08.150794 1822 raft_consensus.cc:482] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:08.150913 1822 raft_consensus.cc:3037] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:08.153141 1822 raft_consensus.cc:504] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: false } }
I20240501 20:09:08.153782 1822 leader_election.cc:288] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d [CANDIDATE]: Term 2 election: Requested vote from peers c0931967d12847e08d9ff883c85f4237 (127.29.219.65:36897)
I20240501 20:09:08.154345 625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a1f02449b91442a897ac2278ee98e29" candidate_uuid: "be64c56dce3d46fa80ba75efff719a7d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c0931967d12847e08d9ff883c85f4237"
I20240501 20:09:08.154596 625 raft_consensus.cc:3032] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:08.154703 625 raft_consensus.cc:729] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0931967d12847e08d9ff883c85f4237, State: Running, Role: LEADER
I20240501 20:09:08.155076 625 consensus_queue.cc:260] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: false } }
I20240501 20:09:08.155313 625 raft_consensus.cc:3037] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:08.157240 625 raft_consensus.cc:2454] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be64c56dce3d46fa80ba75efff719a7d in term 2.
I20240501 20:09:08.157624 1756 leader_election.cc:302] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: be64c56dce3d46fa80ba75efff719a7d, c0931967d12847e08d9ff883c85f4237; no voters:
I20240501 20:09:08.157935 2702 raft_consensus.cc:2781] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:08.158088 2702 raft_consensus.cc:686] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d [term 2 LEADER]: Becoming Leader. State: Replica: be64c56dce3d46fa80ba75efff719a7d, State: Running, Role: LEADER
I20240501 20:09:08.158318 2702 consensus_queue.cc:237] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: false } }
I20240501 20:09:08.159987 758 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "998f81b7cc7e4df7832f4694c4f34731"
dest_uuid: "b811afcdb9d34a38b7e14e693b58093e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38012
I20240501 20:09:08.160207 758 raft_consensus.cc:595] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:08.161293 483 catalog_manager.cc:5484] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d reported cstate change: term changed from 1 to 2, leader changed from c0931967d12847e08d9ff883c85f4237 (127.29.219.65) to be64c56dce3d46fa80ba75efff719a7d (127.29.219.67). New cstate: current_term: 2 leader_uuid: "be64c56dce3d46fa80ba75efff719a7d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:08.222785 2481 raft_consensus.cc:979] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e: : Instructing follower 52541d349ee24edaac72825773fc848a to start an election
I20240501 20:09:08.222988 2717 raft_consensus.cc:1067] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Signalling peer 52541d349ee24edaac72825773fc848a to start an election
I20240501 20:09:08.223675 1954 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "998f81b7cc7e4df7832f4694c4f34731"
dest_uuid: "52541d349ee24edaac72825773fc848a"
from {username='slave'} at 127.29.219.66:57587
I20240501 20:09:08.223925 1954 raft_consensus.cc:482] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:08.224063 1954 raft_consensus.cc:3037] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:08.227061 1954 raft_consensus.cc:504] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: false } }
I20240501 20:09:08.227610 1954 leader_election.cc:288] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a [CANDIDATE]: Term 2 election: Requested vote from peers b811afcdb9d34a38b7e14e693b58093e (127.29.219.66:40067)
I20240501 20:09:08.228183 758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "998f81b7cc7e4df7832f4694c4f34731" candidate_uuid: "52541d349ee24edaac72825773fc848a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b811afcdb9d34a38b7e14e693b58093e"
I20240501 20:09:08.228390 758 raft_consensus.cc:3032] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:08.228484 758 raft_consensus.cc:729] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b811afcdb9d34a38b7e14e693b58093e, State: Running, Role: LEADER
I20240501 20:09:08.228734 758 consensus_queue.cc:260] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: false } }
I20240501 20:09:08.229004 758 raft_consensus.cc:3037] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:08.231757 758 raft_consensus.cc:2454] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52541d349ee24edaac72825773fc848a in term 2.
I20240501 20:09:08.232173 1886 leader_election.cc:302] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 52541d349ee24edaac72825773fc848a, b811afcdb9d34a38b7e14e693b58093e; no voters:
I20240501 20:09:08.232548 2653 raft_consensus.cc:2781] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:08.232724 2653 raft_consensus.cc:686] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a [term 2 LEADER]: Becoming Leader. State: Replica: 52541d349ee24edaac72825773fc848a, State: Running, Role: LEADER
I20240501 20:09:08.232982 2653 consensus_queue.cc:237] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: false } }
I20240501 20:09:08.235463 483 catalog_manager.cc:5484] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a reported cstate change: term changed from 1 to 2, leader changed from b811afcdb9d34a38b7e14e693b58093e (127.29.219.66) to 52541d349ee24edaac72825773fc848a (127.29.219.68). New cstate: current_term: 2 leader_uuid: "52541d349ee24edaac72825773fc848a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:08.276173 758 raft_consensus.cc:1261] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [term 2 FOLLOWER]: Refusing update from remote peer 09cd35a57c9d4c0f86544e6ac1a55279: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:09:08.276903 2899 consensus_queue.cc:1035] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:09:08.334347 625 raft_consensus.cc:1261] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Refusing update from remote peer 52675e825f834cf3ac80a5777be65973: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:09:08.335238 2697 consensus_queue.cc:1035] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:09:08.386014 625 raft_consensus.cc:1261] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Refusing update from remote peer efa05351dbe344c7a451e7fdbeed47f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:09:08.386760 2690 consensus_queue.cc:1035] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:09:08.394222 625 raft_consensus.cc:1261] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Refusing update from remote peer 52541d349ee24edaac72825773fc848a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:09:08.394977 2653 consensus_queue.cc:1035] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:09:08.405951 625 raft_consensus.cc:1261] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Refusing update from remote peer be64c56dce3d46fa80ba75efff719a7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:09:08.406716 2974 consensus_queue.cc:1035] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:09:08.444278 2215 consensus_queue.cc:237] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: false } }
I20240501 20:09:08.445276 3028 raft_consensus.cc:2932] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b811afcdb9d34a38b7e14e693b58093e (127.29.219.66) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: false } } }
I20240501 20:09:08.448969 470 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e606598eee444113b75b26943d884754 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 20:09:08.450261 483 catalog_manager.cc:5484] T e606598eee444113b75b26943d884754 P 09cd35a57c9d4c0f86544e6ac1a55279 reported cstate change: config changed from index 5 to 7, VOTER b811afcdb9d34a38b7e14e693b58093e (127.29.219.66) evicted. New cstate: current_term: 2 leader_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:08.454311 737 tablet_service.cc:1514] Processing DeleteTablet for tablet e606598eee444113b75b26943d884754 with delete_type TABLET_DATA_TOMBSTONED (TS b811afcdb9d34a38b7e14e693b58093e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51662
I20240501 20:09:08.455214 3043 tablet_replica.cc:330] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e: stopping tablet replica
I20240501 20:09:08.455638 3043 raft_consensus.cc:2229] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:08.456229 3043 raft_consensus.cc:2258] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:08.457150 758 raft_consensus.cc:1261] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [term 2 FOLLOWER]: Refusing update from remote peer 52675e825f834cf3ac80a5777be65973: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:09:08.457777 2697 consensus_queue.cc:1035] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:09:08.457937 3043 ts_tablet_manager.cc:1875] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:08.463831 3043 ts_tablet_manager.cc:1888] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:09:08.464015 3043 log.cc:1196] T e606598eee444113b75b26943d884754 P b811afcdb9d34a38b7e14e693b58093e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/e606598eee444113b75b26943d884754
I20240501 20:09:08.464912 468 catalog_manager.cc:4830] TS b811afcdb9d34a38b7e14e693b58093e (127.29.219.66:40067): tablet e606598eee444113b75b26943d884754 (table rebalance_test_table_3 [id=54c7219e95dd4c53bfd6d508026d4771]) successfully deleted
I20240501 20:09:08.472853 1954 consensus_queue.cc:237] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: false } }
I20240501 20:09:08.473865 2913 raft_consensus.cc:2932] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c0931967d12847e08d9ff883c85f4237 (127.29.219.65) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: false } } }
I20240501 20:09:08.476783 471 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 09ac60ecdffa40b2a321c810d7890609 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 20:09:08.478044 482 catalog_manager.cc:5484] T 09ac60ecdffa40b2a321c810d7890609 P 52541d349ee24edaac72825773fc848a reported cstate change: config changed from index 5 to 7, VOTER c0931967d12847e08d9ff883c85f4237 (127.29.219.65) evicted. New cstate: current_term: 2 leader_uuid: "52541d349ee24edaac72825773fc848a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:08.482535 604 tablet_service.cc:1514] Processing DeleteTablet for tablet 09ac60ecdffa40b2a321c810d7890609 with delete_type TABLET_DATA_TOMBSTONED (TS c0931967d12847e08d9ff883c85f4237 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56600
I20240501 20:09:08.483469 3054 tablet_replica.cc:330] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237: stopping tablet replica
I20240501 20:09:08.483945 3054 raft_consensus.cc:2229] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:08.484354 3054 raft_consensus.cc:2258] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:08.485759 3054 ts_tablet_manager.cc:1875] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:08.492887 3054 ts_tablet_manager.cc:1888] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:09:08.493134 3054 log.cc:1196] T 09ac60ecdffa40b2a321c810d7890609 P c0931967d12847e08d9ff883c85f4237: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/09ac60ecdffa40b2a321c810d7890609
I20240501 20:09:08.494156 470 catalog_manager.cc:4830] TS c0931967d12847e08d9ff883c85f4237 (127.29.219.65:36897): tablet 09ac60ecdffa40b2a321c810d7890609 (table rebalance_test_table_2 [id=318aa2fee23548949c6edf6fe8ff8283]) successfully deleted
I20240501 20:09:08.535367 2345 consensus_queue.cc:237] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: false } }
I20240501 20:09:08.536800 2962 raft_consensus.cc:2932] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c0931967d12847e08d9ff883c85f4237 (127.29.219.65) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: false } } }
I20240501 20:09:08.541263 468 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5ae809988743421f8046716158b43a88 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 20:09:08.542297 482 catalog_manager.cc:5484] T 5ae809988743421f8046716158b43a88 P 52675e825f834cf3ac80a5777be65973 reported cstate change: config changed from index 5 to 7, VOTER c0931967d12847e08d9ff883c85f4237 (127.29.219.65) evicted. New cstate: current_term: 2 leader_uuid: "52675e825f834cf3ac80a5777be65973" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:08.545944 604 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae809988743421f8046716158b43a88 with delete_type TABLET_DATA_TOMBSTONED (TS c0931967d12847e08d9ff883c85f4237 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56600
I20240501 20:09:08.546409 3054 tablet_replica.cc:330] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237: stopping tablet replica
I20240501 20:09:08.546653 3054 raft_consensus.cc:2229] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:08.546854 3054 raft_consensus.cc:2258] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:08.547885 3054 ts_tablet_manager.cc:1875] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:08.552752 3054 ts_tablet_manager.cc:1888] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:09:08.552868 3054 log.cc:1196] T 5ae809988743421f8046716158b43a88 P c0931967d12847e08d9ff883c85f4237: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/5ae809988743421f8046716158b43a88
I20240501 20:09:08.553684 470 catalog_manager.cc:4830] TS c0931967d12847e08d9ff883c85f4237 (127.29.219.65:36897): tablet 5ae809988743421f8046716158b43a88 (table rebalance_test_table_2 [id=318aa2fee23548949c6edf6fe8ff8283]) successfully deleted
I20240501 20:09:08.558461 758 raft_consensus.cc:1261] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [term 2 FOLLOWER]: Refusing update from remote peer efa05351dbe344c7a451e7fdbeed47f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:09:08.559216 2690 consensus_queue.cc:1035] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:09:08.567589 2084 consensus_queue.cc:237] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: false } }
I20240501 20:09:08.568478 2690 raft_consensus.cc:2932] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c0931967d12847e08d9ff883c85f4237 (127.29.219.65) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: false } } }
I20240501 20:09:08.568920 625 consensus_queue.cc:237] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: true } }
I20240501 20:09:08.570678 2705 raft_consensus.cc:2932] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 09cd35a57c9d4c0f86544e6ac1a55279 (127.29.219.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: true } } }
I20240501 20:09:08.573510 468 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2ad6ed0144eb4b619964ee4f86b9a842 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 20:09:08.574925 482 catalog_manager.cc:5484] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 reported cstate change: config changed from index -1 to 4, NON_VOTER 09cd35a57c9d4c0f86544e6ac1a55279 (127.29.219.70) added. New cstate: current_term: 1 leader_uuid: "c0931967d12847e08d9ff883c85f4237" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:08.575811 483 catalog_manager.cc:5484] T 2ad6ed0144eb4b619964ee4f86b9a842 P efa05351dbe344c7a451e7fdbeed47f0 reported cstate change: config changed from index 5 to 7, VOTER c0931967d12847e08d9ff883c85f4237 (127.29.219.65) evicted. New cstate: current_term: 2 leader_uuid: "efa05351dbe344c7a451e7fdbeed47f0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:08.579730 604 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ad6ed0144eb4b619964ee4f86b9a842 with delete_type TABLET_DATA_TOMBSTONED (TS c0931967d12847e08d9ff883c85f4237 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56600
I20240501 20:09:08.580070 3054 tablet_replica.cc:330] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237: stopping tablet replica
I20240501 20:09:08.580245 3054 raft_consensus.cc:2229] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:08.580415 3054 raft_consensus.cc:2258] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:08.581221 3054 ts_tablet_manager.cc:1875] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:08.585889 3054 ts_tablet_manager.cc:1888] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:09:08.586047 3054 log.cc:1196] T 2ad6ed0144eb4b619964ee4f86b9a842 P c0931967d12847e08d9ff883c85f4237: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/2ad6ed0144eb4b619964ee4f86b9a842
I20240501 20:09:08.586884 470 catalog_manager.cc:4830] TS c0931967d12847e08d9ff883c85f4237 (127.29.219.65:36897): tablet 2ad6ed0144eb4b619964ee4f86b9a842 (table rebalance_test_table_0 [id=c30633c72e73448b81efc5fc64800aa6]) successfully deleted
W20240501 20:09:08.588259 560 consensus_peers.cc:485] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 -> Peer 09cd35a57c9d4c0f86544e6ac1a55279 (127.29.219.70:40955): Couldn't send request to peer 09cd35a57c9d4c0f86544e6ac1a55279. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2f958611c6c4f459055bd2a7cc75f15. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:08.658494 625 raft_consensus.cc:1261] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Refusing update from remote peer be64c56dce3d46fa80ba75efff719a7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:09:08.659077 2974 consensus_queue.cc:1035] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:09:08.667112 1822 consensus_queue.cc:237] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: false } }
I20240501 20:09:08.668145 3137 raft_consensus.cc:2932] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c0931967d12847e08d9ff883c85f4237 (127.29.219.65) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: false } } }
I20240501 20:09:08.668397 2345 consensus_queue.cc:237] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: false } }
I20240501 20:09:08.669595 2697 raft_consensus.cc:2932] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b811afcdb9d34a38b7e14e693b58093e (127.29.219.66) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: false } } }
I20240501 20:09:08.671272 2084 consensus_queue.cc:237] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: false } }
I20240501 20:09:08.672259 2690 raft_consensus.cc:2932] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b811afcdb9d34a38b7e14e693b58093e (127.29.219.66) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: false } } }
I20240501 20:09:08.673231 469 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f71d46683ee74f0b8adb7ffdc4ec2688 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 20:09:08.674094 468 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0a9d19f112154a65b012072f9d0d4149 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 20:09:08.675545 482 catalog_manager.cc:5484] T 0a9d19f112154a65b012072f9d0d4149 P 52675e825f834cf3ac80a5777be65973 reported cstate change: config changed from index 5 to 7, VOTER b811afcdb9d34a38b7e14e693b58093e (127.29.219.66) evicted. New cstate: current_term: 2 leader_uuid: "52675e825f834cf3ac80a5777be65973" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "52675e825f834cf3ac80a5777be65973" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39829 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:08.677798 468 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2705dcf15dc54cbbb263fcfee0ee8aed with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 20:09:08.678697 483 catalog_manager.cc:5484] T f71d46683ee74f0b8adb7ffdc4ec2688 P be64c56dce3d46fa80ba75efff719a7d reported cstate change: config changed from index 5 to 7, VOTER c0931967d12847e08d9ff883c85f4237 (127.29.219.65) evicted. New cstate: current_term: 2 leader_uuid: "be64c56dce3d46fa80ba75efff719a7d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:08.679865 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a9d19f112154a65b012072f9d0d4149 with delete_type TABLET_DATA_TOMBSTONED (TS b811afcdb9d34a38b7e14e693b58093e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51662
I20240501 20:09:08.680277 3043 tablet_replica.cc:330] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e: stopping tablet replica
I20240501 20:09:08.680462 3043 raft_consensus.cc:2229] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:08.680619 3043 raft_consensus.cc:2258] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:08.681536 3043 ts_tablet_manager.cc:1875] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:08.681934 482 catalog_manager.cc:5484] T 2705dcf15dc54cbbb263fcfee0ee8aed P efa05351dbe344c7a451e7fdbeed47f0 reported cstate change: config changed from index 5 to 7, VOTER b811afcdb9d34a38b7e14e693b58093e (127.29.219.66) evicted. New cstate: current_term: 2 leader_uuid: "efa05351dbe344c7a451e7fdbeed47f0" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "efa05351dbe344c7a451e7fdbeed47f0" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 44913 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:08.683669 604 tablet_service.cc:1514] Processing DeleteTablet for tablet f71d46683ee74f0b8adb7ffdc4ec2688 with delete_type TABLET_DATA_TOMBSTONED (TS c0931967d12847e08d9ff883c85f4237 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56600
I20240501 20:09:08.684201 3054 tablet_replica.cc:330] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237: stopping tablet replica
I20240501 20:09:08.684393 3054 raft_consensus.cc:2229] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:08.684581 3054 raft_consensus.cc:2258] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:08.685994 3054 ts_tablet_manager.cc:1875] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:08.688292 3043 ts_tablet_manager.cc:1888] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:09:08.688421 3043 log.cc:1196] T 0a9d19f112154a65b012072f9d0d4149 P b811afcdb9d34a38b7e14e693b58093e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/0a9d19f112154a65b012072f9d0d4149
I20240501 20:09:08.689294 468 catalog_manager.cc:4830] TS b811afcdb9d34a38b7e14e693b58093e (127.29.219.66:40067): tablet 0a9d19f112154a65b012072f9d0d4149 (table rebalance_test_table_3 [id=54c7219e95dd4c53bfd6d508026d4771]) successfully deleted
I20240501 20:09:08.692723 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 2705dcf15dc54cbbb263fcfee0ee8aed with delete_type TABLET_DATA_TOMBSTONED (TS b811afcdb9d34a38b7e14e693b58093e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51662
I20240501 20:09:08.693081 3043 tablet_replica.cc:330] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e: stopping tablet replica
I20240501 20:09:08.693276 3043 raft_consensus.cc:2229] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:08.693461 3043 raft_consensus.cc:2258] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:08.693935 3054 ts_tablet_manager.cc:1888] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:09:08.694072 3054 log.cc:1196] T f71d46683ee74f0b8adb7ffdc4ec2688 P c0931967d12847e08d9ff883c85f4237: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/f71d46683ee74f0b8adb7ffdc4ec2688
I20240501 20:09:08.694373 3043 ts_tablet_manager.cc:1875] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:08.694902 470 catalog_manager.cc:4830] TS c0931967d12847e08d9ff883c85f4237 (127.29.219.65:36897): tablet f71d46683ee74f0b8adb7ffdc4ec2688 (table rebalance_test_table_4 [id=e27501566faf443498994ced607143d7]) successfully deleted
I20240501 20:09:08.700098 3043 ts_tablet_manager.cc:1888] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:09:08.700232 3043 log.cc:1196] T 2705dcf15dc54cbbb263fcfee0ee8aed P b811afcdb9d34a38b7e14e693b58093e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/2705dcf15dc54cbbb263fcfee0ee8aed
I20240501 20:09:08.701051 468 catalog_manager.cc:4830] TS b811afcdb9d34a38b7e14e693b58093e (127.29.219.66:40067): tablet 2705dcf15dc54cbbb263fcfee0ee8aed (table rebalance_test_table_1 [id=c9860c7b5b694a66abe3f7254149aa83]) successfully deleted
I20240501 20:09:08.753675 758 raft_consensus.cc:1261] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [term 2 FOLLOWER]: Refusing update from remote peer 52541d349ee24edaac72825773fc848a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:09:08.754218 2653 consensus_queue.cc:1035] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:09:08.810725 1822 consensus_queue.cc:237] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: false } }
I20240501 20:09:08.811610 3137 raft_consensus.cc:2932] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c0931967d12847e08d9ff883c85f4237 (127.29.219.65) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: false } } }
I20240501 20:09:08.815229 469 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3a1f02449b91442a897ac2278ee98e29 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 20:09:08.816464 483 catalog_manager.cc:5484] T 3a1f02449b91442a897ac2278ee98e29 P be64c56dce3d46fa80ba75efff719a7d reported cstate change: config changed from index 5 to 7, VOTER c0931967d12847e08d9ff883c85f4237 (127.29.219.65) evicted. New cstate: current_term: 2 leader_uuid: "be64c56dce3d46fa80ba75efff719a7d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "be64c56dce3d46fa80ba75efff719a7d" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42219 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:08.821367 604 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a1f02449b91442a897ac2278ee98e29 with delete_type TABLET_DATA_TOMBSTONED (TS c0931967d12847e08d9ff883c85f4237 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56600
I20240501 20:09:08.821786 3054 tablet_replica.cc:330] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237: stopping tablet replica
I20240501 20:09:08.822007 3054 raft_consensus.cc:2229] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:08.822254 3054 raft_consensus.cc:2258] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:08.823218 3054 ts_tablet_manager.cc:1875] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:08.829082 3054 ts_tablet_manager.cc:1888] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:09:08.829224 3054 log.cc:1196] T 3a1f02449b91442a897ac2278ee98e29 P c0931967d12847e08d9ff883c85f4237: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/3a1f02449b91442a897ac2278ee98e29
I20240501 20:09:08.830122 470 catalog_manager.cc:4830] TS c0931967d12847e08d9ff883c85f4237 (127.29.219.65:36897): tablet 3a1f02449b91442a897ac2278ee98e29 (table rebalance_test_table_0 [id=c30633c72e73448b81efc5fc64800aa6]) successfully deleted
I20240501 20:09:08.836249 1954 consensus_queue.cc:237] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: false } }
I20240501 20:09:08.837227 3180 raft_consensus.cc:2932] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b811afcdb9d34a38b7e14e693b58093e (127.29.219.66) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: false } } }
I20240501 20:09:08.841089 471 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 998f81b7cc7e4df7832f4694c4f34731 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 20:09:08.842401 483 catalog_manager.cc:5484] T 998f81b7cc7e4df7832f4694c4f34731 P 52541d349ee24edaac72825773fc848a reported cstate change: config changed from index 5 to 7, VOTER b811afcdb9d34a38b7e14e693b58093e (127.29.219.66) evicted. New cstate: current_term: 2 leader_uuid: "52541d349ee24edaac72825773fc848a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "52541d349ee24edaac72825773fc848a" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 46431 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:08.846181 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 998f81b7cc7e4df7832f4694c4f34731 with delete_type TABLET_DATA_TOMBSTONED (TS b811afcdb9d34a38b7e14e693b58093e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51662
I20240501 20:09:08.846630 3043 tablet_replica.cc:330] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e: stopping tablet replica
I20240501 20:09:08.846848 3043 raft_consensus.cc:2229] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:08.847079 3043 raft_consensus.cc:2258] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:08.848464 3043 ts_tablet_manager.cc:1875] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:08.856994 3043 ts_tablet_manager.cc:1888] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:09:08.857262 3043 log.cc:1196] T 998f81b7cc7e4df7832f4694c4f34731 P b811afcdb9d34a38b7e14e693b58093e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/998f81b7cc7e4df7832f4694c4f34731
I20240501 20:09:08.858290 468 catalog_manager.cc:4830] TS b811afcdb9d34a38b7e14e693b58093e (127.29.219.66:40067): tablet 998f81b7cc7e4df7832f4694c4f34731 (table rebalance_test_table_1 [id=c9860c7b5b694a66abe3f7254149aa83]) successfully deleted
I20240501 20:09:09.106422 3202 ts_tablet_manager.cc:902] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279: Initiating tablet copy from peer c0931967d12847e08d9ff883c85f4237 (127.29.219.65:36897)
I20240501 20:09:09.108045 3202 tablet_copy_client.cc:308] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.65:36897
I20240501 20:09:09.113750 644 tablet_copy_service.cc:140] P c0931967d12847e08d9ff883c85f4237: Received BeginTabletCopySession request for tablet d2f958611c6c4f459055bd2a7cc75f15 from peer 09cd35a57c9d4c0f86544e6ac1a55279 ({username='slave'} at 127.29.219.70:35957)
I20240501 20:09:09.113915 644 tablet_copy_service.cc:161] P c0931967d12847e08d9ff883c85f4237: Beginning new tablet copy session on tablet d2f958611c6c4f459055bd2a7cc75f15 from peer 09cd35a57c9d4c0f86544e6ac1a55279 at {username='slave'} at 127.29.219.70:35957: session id = 09cd35a57c9d4c0f86544e6ac1a55279-d2f958611c6c4f459055bd2a7cc75f15
I20240501 20:09:09.115873 644 tablet_copy_source_session.cc:205] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:09.117053 3202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2f958611c6c4f459055bd2a7cc75f15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:09.121433 3202 tablet_copy_client.cc:783] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:09.121682 3202 tablet_copy_client.cc:647] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:09.123459 3202 tablet_copy_client.cc:523] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:09.125792 3202 tablet_bootstrap.cc:492] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279: Bootstrap starting.
I20240501 20:09:09.155915 3202 tablet_bootstrap.cc:492] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:09.156345 3202 tablet_bootstrap.cc:492] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279: Bootstrap complete.
I20240501 20:09:09.156599 3202 ts_tablet_manager.cc:1367] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279: Time spent bootstrapping tablet: real 0.031s user 0.019s sys 0.012s
I20240501 20:09:09.157400 3202 raft_consensus.cc:348] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: true } }
I20240501 20:09:09.157603 3202 raft_consensus.cc:729] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09cd35a57c9d4c0f86544e6ac1a55279, State: Initialized, Role: LEARNER
I20240501 20:09:09.157799 3202 consensus_queue.cc:260] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: true } }
I20240501 20:09:09.158468 3202 ts_tablet_manager.cc:1398] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20240501 20:09:09.159389 644 tablet_copy_service.cc:342] P c0931967d12847e08d9ff883c85f4237: Request end of tablet copy session 09cd35a57c9d4c0f86544e6ac1a55279-d2f958611c6c4f459055bd2a7cc75f15 received from {username='slave'} at 127.29.219.70:35957
I20240501 20:09:09.159540 644 tablet_copy_service.cc:433] P c0931967d12847e08d9ff883c85f4237: ending tablet copy session 09cd35a57c9d4c0f86544e6ac1a55279-d2f958611c6c4f459055bd2a7cc75f15 on tablet d2f958611c6c4f459055bd2a7cc75f15 with peer 09cd35a57c9d4c0f86544e6ac1a55279
I20240501 20:09:09.716073 2215 raft_consensus.cc:1203] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:09:10.161010 3228 raft_consensus.cc:1050] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237: attempting to promote NON_VOTER 09cd35a57c9d4c0f86544e6ac1a55279 to VOTER
I20240501 20:09:10.161643 3228 consensus_queue.cc:237] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: false } }
I20240501 20:09:10.163599 2215 raft_consensus.cc:1261] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279 [term 1 LEARNER]: Refusing update from remote peer c0931967d12847e08d9ff883c85f4237: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:09:10.164126 3226 consensus_queue.cc:1035] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:10.166419 3228 raft_consensus.cc:2932] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 09cd35a57c9d4c0f86544e6ac1a55279 (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: false } } }
I20240501 20:09:10.167086 2215 raft_consensus.cc:2932] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 09cd35a57c9d4c0f86544e6ac1a55279 (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: false } } }
I20240501 20:09:10.170753 483 catalog_manager.cc:5484] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 reported cstate change: config changed from index 4 to 5, 09cd35a57c9d4c0f86544e6ac1a55279 (127.29.219.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c0931967d12847e08d9ff883c85f4237" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:10.273813 624 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d2f958611c6c4f459055bd2a7cc75f15"
dest_uuid: "c0931967d12847e08d9ff883c85f4237"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45898
I20240501 20:09:10.274025 624 raft_consensus.cc:595] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:10.493522 624 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d2f958611c6c4f459055bd2a7cc75f15"
dest_uuid: "c0931967d12847e08d9ff883c85f4237"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45920
I20240501 20:09:10.493744 624 raft_consensus.cc:595] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:10.712124 624 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d2f958611c6c4f459055bd2a7cc75f15"
dest_uuid: "c0931967d12847e08d9ff883c85f4237"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45938
I20240501 20:09:10.712318 624 raft_consensus.cc:595] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:10.775072 3276 raft_consensus.cc:979] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237: : Instructing follower 09cd35a57c9d4c0f86544e6ac1a55279 to start an election
I20240501 20:09:10.775251 3276 raft_consensus.cc:1067] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Signalling peer 09cd35a57c9d4c0f86544e6ac1a55279 to start an election
I20240501 20:09:10.776027 2215 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d2f958611c6c4f459055bd2a7cc75f15"
dest_uuid: "09cd35a57c9d4c0f86544e6ac1a55279"
from {username='slave'} at 127.29.219.65:50697
I20240501 20:09:10.776234 2215 raft_consensus.cc:482] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:10.776374 2215 raft_consensus.cc:3037] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:10.778329 2215 raft_consensus.cc:504] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: false } }
I20240501 20:09:10.778973 2215 leader_election.cc:288] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279 [CANDIDATE]: Term 2 election: Requested vote from peers c0931967d12847e08d9ff883c85f4237 (127.29.219.65:36897)
I20240501 20:09:10.779474 624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2f958611c6c4f459055bd2a7cc75f15" candidate_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c0931967d12847e08d9ff883c85f4237"
I20240501 20:09:10.779654 624 raft_consensus.cc:3032] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:10.779757 624 raft_consensus.cc:729] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0931967d12847e08d9ff883c85f4237, State: Running, Role: LEADER
I20240501 20:09:10.780026 624 consensus_queue.cc:260] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: false } }
I20240501 20:09:10.780301 624 raft_consensus.cc:3037] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:10.782884 624 raft_consensus.cc:2454] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09cd35a57c9d4c0f86544e6ac1a55279 in term 2.
I20240501 20:09:10.783407 2149 leader_election.cc:302] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 09cd35a57c9d4c0f86544e6ac1a55279, c0931967d12847e08d9ff883c85f4237; no voters:
I20240501 20:09:10.784009 3278 raft_consensus.cc:2781] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:10.784410 3278 raft_consensus.cc:686] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279 [term 2 LEADER]: Becoming Leader. State: Replica: 09cd35a57c9d4c0f86544e6ac1a55279, State: Running, Role: LEADER
I20240501 20:09:10.784722 3278 consensus_queue.cc:237] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: false } }
I20240501 20:09:10.787016 482 catalog_manager.cc:5484] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279 reported cstate change: term changed from 1 to 2, leader changed from c0931967d12847e08d9ff883c85f4237 (127.29.219.65) to 09cd35a57c9d4c0f86544e6ac1a55279 (127.29.219.70). New cstate: current_term: 2 leader_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:09:10.791803 470 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet d2f958611c6c4f459055bd2a7cc75f15 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 09cd35a57c9d4c0f86544e6ac1a55279 (127.29.219.70:40955) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240501 20:09:11.262380 624 raft_consensus.cc:1261] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Refusing update from remote peer 09cd35a57c9d4c0f86544e6ac1a55279: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:09:11.263109 3278 consensus_queue.cc:1035] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:09:11.417416 2215 consensus_queue.cc:237] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: false } }
I20240501 20:09:11.418284 3278 raft_consensus.cc:2932] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c0931967d12847e08d9ff883c85f4237 (127.29.219.65) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: false } } }
I20240501 20:09:11.421098 470 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d2f958611c6c4f459055bd2a7cc75f15 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 20:09:11.422472 483 catalog_manager.cc:5484] T d2f958611c6c4f459055bd2a7cc75f15 P 09cd35a57c9d4c0f86544e6ac1a55279 reported cstate change: config changed from index 5 to 7, VOTER c0931967d12847e08d9ff883c85f4237 (127.29.219.65) evicted. New cstate: current_term: 2 leader_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "09cd35a57c9d4c0f86544e6ac1a55279" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 40955 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:11.426386 604 tablet_service.cc:1514] Processing DeleteTablet for tablet d2f958611c6c4f459055bd2a7cc75f15 with delete_type TABLET_DATA_TOMBSTONED (TS c0931967d12847e08d9ff883c85f4237 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56600
I20240501 20:09:11.427186 3306 tablet_replica.cc:330] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237: stopping tablet replica
I20240501 20:09:11.427356 3306 raft_consensus.cc:2229] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:11.427548 3306 raft_consensus.cc:2258] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:11.428432 3306 ts_tablet_manager.cc:1875] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:11.433277 3306 ts_tablet_manager.cc:1888] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:09:11.433389 3306 log.cc:1196] T d2f958611c6c4f459055bd2a7cc75f15 P c0931967d12847e08d9ff883c85f4237: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/d2f958611c6c4f459055bd2a7cc75f15
I20240501 20:09:11.434191 470 catalog_manager.cc:4830] TS c0931967d12847e08d9ff883c85f4237 (127.29.219.65:36897): tablet d2f958611c6c4f459055bd2a7cc75f15 (table rebalance_test_table_4 [id=e27501566faf443498994ced607143d7]) successfully deleted
I20240501 20:09:11.661275 2195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:11.663940 1933 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:11.685909 2326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:11.720594 737 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:11.721185 1802 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:11.739560 604 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:11.745431 2064 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:11.900665 2195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:11.924449 2326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:11.927117 1933 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:11.928037 737 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:11.936293 1802 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:11.959973 2064 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:11.961076 604 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:12.672796 2195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:12.677855 1933 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:12.682104 737 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:12.692713 2326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:12.714085 2064 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:12.742506 604 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:12.745771 1802 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:12.927590 2195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:12.934281 1933 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:12.943701 2326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:12.949416 737 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:12.966300 2064 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:12.966322 604 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:13.002390 1802 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:13.269107 2195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:13.302335 2326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:13.304054 1933 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:13.307457 737 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:13.312886 604 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:13.320188 1802 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:13.328608 2064 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ef92b2d391194ecf9cfbe91fe77708c7 | 127.29.219.126:46839 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.219.84:42611 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
09cd35a57c9d4c0f86544e6ac1a55279 | 127.29.219.70:40955 | HEALTHY | <none> | 2 | 0
52541d349ee24edaac72825773fc848a | 127.29.219.68:46431 | HEALTHY | <none> | 2 | 0
52675e825f834cf3ac80a5777be65973 | 127.29.219.71:39829 | HEALTHY | <none> | 2 | 0
b811afcdb9d34a38b7e14e693b58093e | 127.29.219.66:40067 | HEALTHY | <none> | 3 | 0
be64c56dce3d46fa80ba75efff719a7d | 127.29.219.67:42219 | HEALTHY | <none> | 2 | 0
c0931967d12847e08d9ff883c85f4237 | 127.29.219.65:36897 | HEALTHY | <none> | 3 | 0
efa05351dbe344c7a451e7fdbeed47f0 | 127.29.219.69:44913 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.29.219.65 | experimental | 127.29.219.65:36897
local_ip_for_outbound_sockets | 127.29.219.66 | experimental | 127.29.219.66:40067
local_ip_for_outbound_sockets | 127.29.219.67 | experimental | 127.29.219.67:42219
local_ip_for_outbound_sockets | 127.29.219.68 | experimental | 127.29.219.68:46431
local_ip_for_outbound_sockets | 127.29.219.69 | experimental | 127.29.219.69:44913
local_ip_for_outbound_sockets | 127.29.219.70 | experimental | 127.29.219.70:40955
local_ip_for_outbound_sockets | 127.29.219.71 | experimental | 127.29.219.71:39829
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.219.65:36897
server_dump_info_path | /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.219.66:40067
server_dump_info_path | /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.219.67:42219
server_dump_info_path | /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.29.219.68:46431
server_dump_info_path | /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.219.69:44913
server_dump_info_path | /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.219.70:40955
server_dump_info_path | /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.29.219.71:39829
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.219.84:42611 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 20:09:13.400245 30573 log_verifier.cc:126] Checking tablet 09ac60ecdffa40b2a321c810d7890609
I20240501 20:09:13.416249 30573 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 09ac60ecdffa40b2a321c810d7890609
I20240501 20:09:13.416411 30573 log_verifier.cc:126] Checking tablet 0a9d19f112154a65b012072f9d0d4149
I20240501 20:09:13.429229 30573 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 0a9d19f112154a65b012072f9d0d4149
I20240501 20:09:13.429364 30573 log_verifier.cc:126] Checking tablet 1b76d484a0a7446081d3e7dde20af484
I20240501 20:09:13.441675 30573 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1b76d484a0a7446081d3e7dde20af484
I20240501 20:09:13.441808 30573 log_verifier.cc:126] Checking tablet 1ef1cdbd7aa34985adb56e32c6eb64df
I20240501 20:09:13.449404 30573 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 1ef1cdbd7aa34985adb56e32c6eb64df
I20240501 20:09:13.449620 30573 log_verifier.cc:126] Checking tablet 2705dcf15dc54cbbb263fcfee0ee8aed
I20240501 20:09:13.462302 30573 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 2705dcf15dc54cbbb263fcfee0ee8aed
I20240501 20:09:13.462411 30573 log_verifier.cc:126] Checking tablet 2ad6ed0144eb4b619964ee4f86b9a842
I20240501 20:09:13.475055 30573 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 2ad6ed0144eb4b619964ee4f86b9a842
I20240501 20:09:13.475173 30573 log_verifier.cc:126] Checking tablet 3a1f02449b91442a897ac2278ee98e29
I20240501 20:09:13.487660 30573 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 3a1f02449b91442a897ac2278ee98e29
I20240501 20:09:13.487790 30573 log_verifier.cc:126] Checking tablet 5786820dd3d84b86a7a800b034a3e833
I20240501 20:09:13.501706 30573 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 5786820dd3d84b86a7a800b034a3e833
I20240501 20:09:13.501832 30573 log_verifier.cc:126] Checking tablet 5ae809988743421f8046716158b43a88
I20240501 20:09:13.516168 30573 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 5ae809988743421f8046716158b43a88
I20240501 20:09:13.516301 30573 log_verifier.cc:126] Checking tablet 6cee76b4a3ef4e7a87714c210cf705ac
I20240501 20:09:13.530295 30573 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6cee76b4a3ef4e7a87714c210cf705ac
I20240501 20:09:13.530431 30573 log_verifier.cc:126] Checking tablet 76cd9ee2deb14a5aa9bb280b0a8dec53
I20240501 20:09:13.544598 30573 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 76cd9ee2deb14a5aa9bb280b0a8dec53
I20240501 20:09:13.544728 30573 log_verifier.cc:126] Checking tablet 998f81b7cc7e4df7832f4694c4f34731
I20240501 20:09:13.559909 30573 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 998f81b7cc7e4df7832f4694c4f34731
I20240501 20:09:13.560039 30573 log_verifier.cc:126] Checking tablet d2f958611c6c4f459055bd2a7cc75f15
I20240501 20:09:13.574607 30573 log_verifier.cc:177] Verified matching terms for 7 ops in tablet d2f958611c6c4f459055bd2a7cc75f15
I20240501 20:09:13.574791 30573 log_verifier.cc:126] Checking tablet e4b484eeb8424dbb9844f6b96fdacd34
I20240501 20:09:13.588596 30573 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e4b484eeb8424dbb9844f6b96fdacd34
I20240501 20:09:13.588721 30573 log_verifier.cc:126] Checking tablet e606598eee444113b75b26943d884754
I20240501 20:09:13.603618 30573 log_verifier.cc:177] Verified matching terms for 7 ops in tablet e606598eee444113b75b26943d884754
I20240501 20:09:13.603749 30573 log_verifier.cc:126] Checking tablet f71d46683ee74f0b8adb7ffdc4ec2688
I20240501 20:09:13.618126 30573 log_verifier.cc:177] Verified matching terms for 7 ops in tablet f71d46683ee74f0b8adb7ffdc4ec2688
I20240501 20:09:13.621151 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.72:0
--local_ip_for_outbound_sockets=127.29.219.72
--webserver_interface=127.29.219.72
--webserver_port=0
--tserver_master_addrs=127.29.219.126:46839
--builtin_ntp_servers=127.29.219.84:42611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:09:13.990875 3498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:13.991175 3498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:13.991295 3498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:13.999377 3498 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:13.999547 3498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:13.999716 3498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.72
I20240501 20:09:14.009567 3498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:42611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.29.219.72
--webserver_port=0
--tserver_master_addrs=127.29.219.126:46839
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.72
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:14.010965 3498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:14.012701 3498 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:14.026538 3504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:14.026798 3503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:14.027532 3506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:14.028754 3498 server_base.cc:1034] running on GCE node
I20240501 20:09:14.029481 3498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:14.030675 3498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:14.031898 3498 hybrid_clock.cc:648] HybridClock initialized: now 1714594154031832 us; error 56 us; skew 500 ppm
I20240501 20:09:14.032255 3498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:14.040287 3498 webserver.cc:460] Webserver started at http://127.29.219.72:42333/ using document root <none> and password file <none>
I20240501 20:09:14.041100 3498 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:14.041198 3498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:14.041476 3498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:14.044152 3498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "ba0b9e9ac4b848e4b412d78edd507019"
format_stamp: "Formatted at 2024-05-01 20:09:14 on dist-test-slave-h2cj"
I20240501 20:09:14.044852 3498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "ba0b9e9ac4b848e4b412d78edd507019"
format_stamp: "Formatted at 2024-05-01 20:09:14 on dist-test-slave-h2cj"
I20240501 20:09:14.050136 3498 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240501 20:09:14.054116 3512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:14.055488 3498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:14.055644 3498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "ba0b9e9ac4b848e4b412d78edd507019"
format_stamp: "Formatted at 2024-05-01 20:09:14 on dist-test-slave-h2cj"
I20240501 20:09:14.055889 3498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:14.077080 3498 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:14.077946 3498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:14.078226 3498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:14.079782 3498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:14.081239 3498 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:14.081342 3498 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:14.081459 3498 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:14.081537 3498 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:14.125468 3498 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.72:46159
I20240501 20:09:14.125555 3624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.72:46159 every 8 connection(s)
I20240501 20:09:14.126878 3498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 20:09:14.132524 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 3498
I20240501 20:09:14.132723 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 20:09:14.145560 3625 heartbeater.cc:347] Connected to a master server at 127.29.219.126:46839
I20240501 20:09:14.145885 3625 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:14.146780 3625 heartbeater.cc:511] Master 127.29.219.126:46839 requested a full tablet report, sending...
I20240501 20:09:14.148358 475 ts_manager.cc:194] Registered new tserver with Master: ba0b9e9ac4b848e4b412d78edd507019 (127.29.219.72:46159)
I20240501 20:09:14.149768 475 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.72:45719
I20240501 20:09:14.646282 2195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:14.652276 2326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:14.670435 1933 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:14.682206 737 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:14.702474 604 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:14.710784 3560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:14.715106 2064 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:14.724112 1802 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:14.931488 1933 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:14.932240 2195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:14.943758 1802 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:14.964885 2326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:14.967224 604 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:14.980149 3560 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:09:14.980839 737 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:09:14.999224 2064 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:15.085775 758 consensus_queue.cc:237] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: true } }
I20240501 20:09:15.087549 3718 raft_consensus.cc:2932] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ba0b9e9ac4b848e4b412d78edd507019 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: true } } }
I20240501 20:09:15.093662 476 catalog_manager.cc:5484] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e reported cstate change: config changed from index -1 to 4, NON_VOTER ba0b9e9ac4b848e4b412d78edd507019 (127.29.219.72) added. New cstate: current_term: 1 leader_uuid: "b811afcdb9d34a38b7e14e693b58093e" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:15.098901 690 consensus_peers.cc:485] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e -> Peer ba0b9e9ac4b848e4b412d78edd507019 (127.29.219.72:46159): Couldn't send request to peer ba0b9e9ac4b848e4b412d78edd507019. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b76d484a0a7446081d3e7dde20af484. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:15.107074 624 consensus_queue.cc:237] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: true } }
I20240501 20:09:15.109023 3731 raft_consensus.cc:2932] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ba0b9e9ac4b848e4b412d78edd507019 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: true } } }
I20240501 20:09:15.115049 476 catalog_manager.cc:5484] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 reported cstate change: config changed from index -1 to 4, NON_VOTER ba0b9e9ac4b848e4b412d78edd507019 (127.29.219.72) added. New cstate: current_term: 1 leader_uuid: "c0931967d12847e08d9ff883c85f4237" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:15.118428 558 consensus_peers.cc:485] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 -> Peer ba0b9e9ac4b848e4b412d78edd507019 (127.29.219.72:46159): Couldn't send request to peer ba0b9e9ac4b848e4b412d78edd507019. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6cee76b4a3ef4e7a87714c210cf705ac. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:09:15.152416 3625 heartbeater.cc:503] Master 127.29.219.126:46839 was elected leader, sending a full tablet report...
I20240501 20:09:15.491030 3754 ts_tablet_manager.cc:902] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019: Initiating tablet copy from peer c0931967d12847e08d9ff883c85f4237 (127.29.219.65:36897)
I20240501 20:09:15.493875 3754 tablet_copy_client.cc:308] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.65:36897
I20240501 20:09:15.500516 644 tablet_copy_service.cc:140] P c0931967d12847e08d9ff883c85f4237: Received BeginTabletCopySession request for tablet 6cee76b4a3ef4e7a87714c210cf705ac from peer ba0b9e9ac4b848e4b412d78edd507019 ({username='slave'} at 127.29.219.72:60563)
I20240501 20:09:15.500689 644 tablet_copy_service.cc:161] P c0931967d12847e08d9ff883c85f4237: Beginning new tablet copy session on tablet 6cee76b4a3ef4e7a87714c210cf705ac from peer ba0b9e9ac4b848e4b412d78edd507019 at {username='slave'} at 127.29.219.72:60563: session id = ba0b9e9ac4b848e4b412d78edd507019-6cee76b4a3ef4e7a87714c210cf705ac
I20240501 20:09:15.502590 644 tablet_copy_source_session.cc:205] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:15.507176 3754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cee76b4a3ef4e7a87714c210cf705ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:15.509115 3760 ts_tablet_manager.cc:902] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019: Initiating tablet copy from peer b811afcdb9d34a38b7e14e693b58093e (127.29.219.66:40067)
I20240501 20:09:15.511477 3760 tablet_copy_client.cc:308] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.66:40067
I20240501 20:09:15.517446 778 tablet_copy_service.cc:140] P b811afcdb9d34a38b7e14e693b58093e: Received BeginTabletCopySession request for tablet 1b76d484a0a7446081d3e7dde20af484 from peer ba0b9e9ac4b848e4b412d78edd507019 ({username='slave'} at 127.29.219.72:60503)
I20240501 20:09:15.517601 778 tablet_copy_service.cc:161] P b811afcdb9d34a38b7e14e693b58093e: Beginning new tablet copy session on tablet 1b76d484a0a7446081d3e7dde20af484 from peer ba0b9e9ac4b848e4b412d78edd507019 at {username='slave'} at 127.29.219.72:60503: session id = ba0b9e9ac4b848e4b412d78edd507019-1b76d484a0a7446081d3e7dde20af484
I20240501 20:09:15.519490 3754 tablet_copy_client.cc:783] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:15.519646 778 tablet_copy_source_session.cc:205] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:09:15.519863 3754 tablet_copy_client.cc:647] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:15.520987 3760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b76d484a0a7446081d3e7dde20af484. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:15.522692 3754 tablet_copy_client.cc:523] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:15.525933 3754 tablet_bootstrap.cc:492] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019: Bootstrap starting.
I20240501 20:09:15.526087 3760 tablet_copy_client.cc:783] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019: tablet copy: Starting download of 0 data blocks...
I20240501 20:09:15.526331 3760 tablet_copy_client.cc:647] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019: tablet copy: Starting download of 1 WAL segments...
I20240501 20:09:15.528352 3760 tablet_copy_client.cc:523] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:09:15.530848 3760 tablet_bootstrap.cc:492] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019: Bootstrap starting.
I20240501 20:09:15.537245 3760 log.cc:824] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:15.580421 3754 tablet_bootstrap.cc:492] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:15.581024 3754 tablet_bootstrap.cc:492] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019: Bootstrap complete.
I20240501 20:09:15.581501 3754 ts_tablet_manager.cc:1367] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019: Time spent bootstrapping tablet: real 0.056s user 0.048s sys 0.005s
I20240501 20:09:15.582320 3760 tablet_bootstrap.cc:492] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:09:15.582836 3760 tablet_bootstrap.cc:492] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019: Bootstrap complete.
I20240501 20:09:15.583137 3760 ts_tablet_manager.cc:1367] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019: Time spent bootstrapping tablet: real 0.052s user 0.043s sys 0.008s
I20240501 20:09:15.587399 3754 raft_consensus.cc:348] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: true } }
I20240501 20:09:15.587790 3754 raft_consensus.cc:729] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ba0b9e9ac4b848e4b412d78edd507019, State: Initialized, Role: LEARNER
I20240501 20:09:15.588212 3754 consensus_queue.cc:260] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: true } }
I20240501 20:09:15.588644 3760 raft_consensus.cc:348] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: true } }
I20240501 20:09:15.588790 3760 raft_consensus.cc:729] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ba0b9e9ac4b848e4b412d78edd507019, State: Initialized, Role: LEARNER
I20240501 20:09:15.588985 3760 consensus_queue.cc:260] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: true } }
I20240501 20:09:15.590071 3760 ts_tablet_manager.cc:1398] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 20:09:15.590770 3754 ts_tablet_manager.cc:1398] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240501 20:09:15.590879 778 tablet_copy_service.cc:342] P b811afcdb9d34a38b7e14e693b58093e: Request end of tablet copy session ba0b9e9ac4b848e4b412d78edd507019-1b76d484a0a7446081d3e7dde20af484 received from {username='slave'} at 127.29.219.72:60503
I20240501 20:09:15.591058 778 tablet_copy_service.cc:433] P b811afcdb9d34a38b7e14e693b58093e: ending tablet copy session ba0b9e9ac4b848e4b412d78edd507019-1b76d484a0a7446081d3e7dde20af484 on tablet 1b76d484a0a7446081d3e7dde20af484 with peer ba0b9e9ac4b848e4b412d78edd507019
I20240501 20:09:15.591495 644 tablet_copy_service.cc:342] P c0931967d12847e08d9ff883c85f4237: Request end of tablet copy session ba0b9e9ac4b848e4b412d78edd507019-6cee76b4a3ef4e7a87714c210cf705ac received from {username='slave'} at 127.29.219.72:60563
I20240501 20:09:15.591784 644 tablet_copy_service.cc:433] P c0931967d12847e08d9ff883c85f4237: ending tablet copy session ba0b9e9ac4b848e4b412d78edd507019-6cee76b4a3ef4e7a87714c210cf705ac on tablet 6cee76b4a3ef4e7a87714c210cf705ac with peer ba0b9e9ac4b848e4b412d78edd507019
I20240501 20:09:15.982138 3580 raft_consensus.cc:1203] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:09:16.009194 3580 raft_consensus.cc:1203] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:09:16.466753 3790 raft_consensus.cc:1050] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e: attempting to promote NON_VOTER ba0b9e9ac4b848e4b412d78edd507019 to VOTER
I20240501 20:09:16.467389 3790 consensus_queue.cc:237] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: false } }
I20240501 20:09:16.469516 3580 raft_consensus.cc:1261] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019 [term 1 LEARNER]: Refusing update from remote peer b811afcdb9d34a38b7e14e693b58093e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:09:16.470252 3792 consensus_queue.cc:1035] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:16.473708 3790 raft_consensus.cc:2932] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ba0b9e9ac4b848e4b412d78edd507019 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: false } } }
I20240501 20:09:16.474370 3580 raft_consensus.cc:2932] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ba0b9e9ac4b848e4b412d78edd507019 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: false } } }
I20240501 20:09:16.477787 476 catalog_manager.cc:5484] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e reported cstate change: config changed from index 4 to 5, ba0b9e9ac4b848e4b412d78edd507019 (127.29.219.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b811afcdb9d34a38b7e14e693b58093e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:16.478701 3789 raft_consensus.cc:1050] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237: attempting to promote NON_VOTER ba0b9e9ac4b848e4b412d78edd507019 to VOTER
I20240501 20:09:16.479287 3789 consensus_queue.cc:237] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: false } }
I20240501 20:09:16.481806 3580 raft_consensus.cc:1261] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019 [term 1 LEARNER]: Refusing update from remote peer c0931967d12847e08d9ff883c85f4237: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:09:16.482300 3788 consensus_queue.cc:1035] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:09:16.484786 3789 raft_consensus.cc:2932] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ba0b9e9ac4b848e4b412d78edd507019 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: false } } }
I20240501 20:09:16.485292 3580 raft_consensus.cc:2932] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ba0b9e9ac4b848e4b412d78edd507019 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: false } } }
I20240501 20:09:16.489058 482 catalog_manager.cc:5484] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 reported cstate change: config changed from index 4 to 5, ba0b9e9ac4b848e4b412d78edd507019 (127.29.219.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c0931967d12847e08d9ff883c85f4237" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:16.686921 624 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6cee76b4a3ef4e7a87714c210cf705ac"
dest_uuid: "c0931967d12847e08d9ff883c85f4237"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46124
I20240501 20:09:16.687130 624 raft_consensus.cc:595] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:16.704820 758 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1b76d484a0a7446081d3e7dde20af484"
dest_uuid: "b811afcdb9d34a38b7e14e693b58093e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46018
I20240501 20:09:16.705044 758 raft_consensus.cc:595] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:16.923094 624 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6cee76b4a3ef4e7a87714c210cf705ac"
dest_uuid: "c0931967d12847e08d9ff883c85f4237"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46146
I20240501 20:09:16.923296 624 raft_consensus.cc:595] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:16.939960 758 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1b76d484a0a7446081d3e7dde20af484"
dest_uuid: "b811afcdb9d34a38b7e14e693b58093e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46040
I20240501 20:09:16.940218 758 raft_consensus.cc:595] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Received request to transfer leadership
I20240501 20:09:16.997454 3872 raft_consensus.cc:979] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237: : Instructing follower ba0b9e9ac4b848e4b412d78edd507019 to start an election
I20240501 20:09:16.997654 3871 raft_consensus.cc:1067] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Signalling peer ba0b9e9ac4b848e4b412d78edd507019 to start an election
I20240501 20:09:16.998337 3580 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6cee76b4a3ef4e7a87714c210cf705ac"
dest_uuid: "ba0b9e9ac4b848e4b412d78edd507019"
from {username='slave'} at 127.29.219.65:36513
I20240501 20:09:16.998605 3580 raft_consensus.cc:482] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:16.998801 3580 raft_consensus.cc:3037] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:17.000774 3580 raft_consensus.cc:504] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: false } }
I20240501 20:09:17.001966 3580 leader_election.cc:288] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019 [CANDIDATE]: Term 2 election: Requested vote from peers c0931967d12847e08d9ff883c85f4237 (127.29.219.65:36897)
I20240501 20:09:17.002465 624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cee76b4a3ef4e7a87714c210cf705ac" candidate_uuid: "ba0b9e9ac4b848e4b412d78edd507019" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c0931967d12847e08d9ff883c85f4237"
I20240501 20:09:17.002655 624 raft_consensus.cc:3032] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:17.002754 624 raft_consensus.cc:729] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0931967d12847e08d9ff883c85f4237, State: Running, Role: LEADER
I20240501 20:09:17.003005 624 consensus_queue.cc:260] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: false } }
I20240501 20:09:17.003322 624 raft_consensus.cc:3037] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:17.005137 624 raft_consensus.cc:2454] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0b9e9ac4b848e4b412d78edd507019 in term 2.
I20240501 20:09:17.005625 3515 leader_election.cc:302] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ba0b9e9ac4b848e4b412d78edd507019, c0931967d12847e08d9ff883c85f4237; no voters:
I20240501 20:09:17.006279 3873 raft_consensus.cc:2781] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:17.006608 3873 raft_consensus.cc:686] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019 [term 2 LEADER]: Becoming Leader. State: Replica: ba0b9e9ac4b848e4b412d78edd507019, State: Running, Role: LEADER
I20240501 20:09:17.007105 3873 consensus_queue.cc:237] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: false } }
I20240501 20:09:17.010057 482 catalog_manager.cc:5484] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019 reported cstate change: term changed from 1 to 2, leader changed from c0931967d12847e08d9ff883c85f4237 (127.29.219.65) to ba0b9e9ac4b848e4b412d78edd507019 (127.29.219.72). New cstate: current_term: 2 leader_uuid: "ba0b9e9ac4b848e4b412d78edd507019" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:09:17.020565 468 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 6cee76b4a3ef4e7a87714c210cf705ac with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader ba0b9e9ac4b848e4b412d78edd507019 (127.29.219.72:46159) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240501 20:09:17.087669 3880 raft_consensus.cc:979] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e: : Instructing follower ba0b9e9ac4b848e4b412d78edd507019 to start an election
I20240501 20:09:17.087822 3880 raft_consensus.cc:1067] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Signalling peer ba0b9e9ac4b848e4b412d78edd507019 to start an election
I20240501 20:09:17.088487 3580 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1b76d484a0a7446081d3e7dde20af484"
dest_uuid: "ba0b9e9ac4b848e4b412d78edd507019"
from {username='slave'} at 127.29.219.66:54313
I20240501 20:09:17.088738 3580 raft_consensus.cc:482] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:09:17.088830 3580 raft_consensus.cc:3037] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:17.091038 3580 raft_consensus.cc:504] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: false } }
I20240501 20:09:17.091615 3580 leader_election.cc:288] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019 [CANDIDATE]: Term 2 election: Requested vote from peers b811afcdb9d34a38b7e14e693b58093e (127.29.219.66:40067)
I20240501 20:09:17.092108 758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b76d484a0a7446081d3e7dde20af484" candidate_uuid: "ba0b9e9ac4b848e4b412d78edd507019" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "b811afcdb9d34a38b7e14e693b58093e"
I20240501 20:09:17.092280 758 raft_consensus.cc:3032] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:09:17.092408 758 raft_consensus.cc:729] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b811afcdb9d34a38b7e14e693b58093e, State: Running, Role: LEADER
I20240501 20:09:17.092654 758 consensus_queue.cc:260] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: false } }
I20240501 20:09:17.092937 758 raft_consensus.cc:3037] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:09:17.095423 758 raft_consensus.cc:2454] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba0b9e9ac4b848e4b412d78edd507019 in term 2.
I20240501 20:09:17.095955 3513 leader_election.cc:302] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b811afcdb9d34a38b7e14e693b58093e, ba0b9e9ac4b848e4b412d78edd507019; no voters:
I20240501 20:09:17.096355 3873 raft_consensus.cc:2781] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:09:17.096503 3873 raft_consensus.cc:686] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019 [term 2 LEADER]: Becoming Leader. State: Replica: ba0b9e9ac4b848e4b412d78edd507019, State: Running, Role: LEADER
I20240501 20:09:17.096771 3873 consensus_queue.cc:237] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: false } }
I20240501 20:09:17.099583 482 catalog_manager.cc:5484] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019 reported cstate change: term changed from 1 to 2, leader changed from b811afcdb9d34a38b7e14e693b58093e (127.29.219.66) to ba0b9e9ac4b848e4b412d78edd507019 (127.29.219.72). New cstate: current_term: 2 leader_uuid: "ba0b9e9ac4b848e4b412d78edd507019" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:17.476469 624 raft_consensus.cc:1261] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Refusing update from remote peer ba0b9e9ac4b848e4b412d78edd507019: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:09:17.477222 3873 consensus_queue.cc:1035] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0931967d12847e08d9ff883c85f4237" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 36897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:09:17.561316 758 raft_consensus.cc:1261] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [term 2 FOLLOWER]: Refusing update from remote peer ba0b9e9ac4b848e4b412d78edd507019: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:09:17.561941 3874 consensus_queue.cc:1035] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b811afcdb9d34a38b7e14e693b58093e" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:09:17.651270 3579 consensus_queue.cc:237] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: false } }
I20240501 20:09:17.652139 3874 raft_consensus.cc:2932] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c0931967d12847e08d9ff883c85f4237 (127.29.219.65) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: false } } }
I20240501 20:09:17.654851 468 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6cee76b4a3ef4e7a87714c210cf705ac with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 20:09:17.656175 476 catalog_manager.cc:5484] T 6cee76b4a3ef4e7a87714c210cf705ac P ba0b9e9ac4b848e4b412d78edd507019 reported cstate change: config changed from index 5 to 7, VOTER c0931967d12847e08d9ff883c85f4237 (127.29.219.65) evicted. New cstate: current_term: 2 leader_uuid: "ba0b9e9ac4b848e4b412d78edd507019" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:17.660187 604 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cee76b4a3ef4e7a87714c210cf705ac with delete_type TABLET_DATA_TOMBSTONED (TS c0931967d12847e08d9ff883c85f4237 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56600
I20240501 20:09:17.660708 3919 tablet_replica.cc:330] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237: stopping tablet replica
I20240501 20:09:17.660908 3919 raft_consensus.cc:2229] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:17.661113 3919 raft_consensus.cc:2258] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:17.662631 3919 ts_tablet_manager.cc:1875] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:17.666956 3919 ts_tablet_manager.cc:1888] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:09:17.667062 3919 log.cc:1196] T 6cee76b4a3ef4e7a87714c210cf705ac P c0931967d12847e08d9ff883c85f4237: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/6cee76b4a3ef4e7a87714c210cf705ac
I20240501 20:09:17.667816 470 catalog_manager.cc:4830] TS c0931967d12847e08d9ff883c85f4237 (127.29.219.65:36897): tablet 6cee76b4a3ef4e7a87714c210cf705ac (table rebalance_test_table_4 [id=e27501566faf443498994ced607143d7]) successfully deleted
I20240501 20:09:17.724835 3579 consensus_queue.cc:237] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: false } }
I20240501 20:09:17.725656 3873 raft_consensus.cc:2932] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER b811afcdb9d34a38b7e14e693b58093e (127.29.219.66) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: false } } }
I20240501 20:09:17.728336 468 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1b76d484a0a7446081d3e7dde20af484 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240501 20:09:17.729400 476 catalog_manager.cc:5484] T 1b76d484a0a7446081d3e7dde20af484 P ba0b9e9ac4b848e4b412d78edd507019 reported cstate change: config changed from index 5 to 7, VOTER b811afcdb9d34a38b7e14e693b58093e (127.29.219.66) evicted. New cstate: current_term: 2 leader_uuid: "ba0b9e9ac4b848e4b412d78edd507019" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ba0b9e9ac4b848e4b412d78edd507019" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 46159 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:09:17.732463 737 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b76d484a0a7446081d3e7dde20af484 with delete_type TABLET_DATA_TOMBSTONED (TS b811afcdb9d34a38b7e14e693b58093e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51662
I20240501 20:09:17.733048 3920 tablet_replica.cc:330] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e: stopping tablet replica
I20240501 20:09:17.733199 3920 raft_consensus.cc:2229] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:17.733350 3920 raft_consensus.cc:2258] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:17.734905 3920 ts_tablet_manager.cc:1875] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:09:17.739188 3920 ts_tablet_manager.cc:1888] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:09:17.739286 3920 log.cc:1196] T 1b76d484a0a7446081d3e7dde20af484 P b811afcdb9d34a38b7e14e693b58093e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/1b76d484a0a7446081d3e7dde20af484
I20240501 20:09:17.739993 468 catalog_manager.cc:4830] TS b811afcdb9d34a38b7e14e693b58093e (127.29.219.66:40067): tablet 1b76d484a0a7446081d3e7dde20af484 (table rebalance_test_table_3 [id=54c7219e95dd4c53bfd6d508026d4771]) successfully deleted
I20240501 20:09:17.920732 2195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:17.931061 1933 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:17.953339 737 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:17.963079 1802 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:17.979249 3560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:17.979334 2064 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:17.980185 2326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:17.985707 604 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:18.169286 2195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:18.176924 1933 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:18.177402 737 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:18.182818 1802 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:18.187206 3560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:18.202059 2326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:18.219251 2064 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:18.225875 604 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:18.525921 1933 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:18.536091 2195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:18.538420 2326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:18.543706 3560 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:18.560463 737 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:18.560592 1802 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:18.604647 604 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:09:18.617096 2064 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
ef92b2d391194ecf9cfbe91fe77708c7 | 127.29.219.126:46839 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.219.84:42611 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
09cd35a57c9d4c0f86544e6ac1a55279 | 127.29.219.70:40955 | HEALTHY | <none> | 2 | 0
52541d349ee24edaac72825773fc848a | 127.29.219.68:46431 | HEALTHY | <none> | 2 | 0
52675e825f834cf3ac80a5777be65973 | 127.29.219.71:39829 | HEALTHY | <none> | 2 | 0
b811afcdb9d34a38b7e14e693b58093e | 127.29.219.66:40067 | HEALTHY | <none> | 2 | 0
ba0b9e9ac4b848e4b412d78edd507019 | 127.29.219.72:46159 | HEALTHY | <none> | 2 | 0
be64c56dce3d46fa80ba75efff719a7d | 127.29.219.67:42219 | HEALTHY | <none> | 2 | 0
c0931967d12847e08d9ff883c85f4237 | 127.29.219.65:36897 | HEALTHY | <none> | 2 | 0
efa05351dbe344c7a451e7fdbeed47f0 | 127.29.219.69:44913 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.29.219.65 | experimental | 127.29.219.65:36897
local_ip_for_outbound_sockets | 127.29.219.66 | experimental | 127.29.219.66:40067
local_ip_for_outbound_sockets | 127.29.219.67 | experimental | 127.29.219.67:42219
local_ip_for_outbound_sockets | 127.29.219.68 | experimental | 127.29.219.68:46431
local_ip_for_outbound_sockets | 127.29.219.69 | experimental | 127.29.219.69:44913
local_ip_for_outbound_sockets | 127.29.219.70 | experimental | 127.29.219.70:40955
local_ip_for_outbound_sockets | 127.29.219.71 | experimental | 127.29.219.71:39829
local_ip_for_outbound_sockets | 127.29.219.72 | experimental | 127.29.219.72:46159
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.219.65:36897
server_dump_info_path | /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.219.66:40067
server_dump_info_path | /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.219.67:42219
server_dump_info_path | /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.29.219.68:46431
server_dump_info_path | /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.219.69:44913
server_dump_info_path | /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.219.70:40955
server_dump_info_path | /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.29.219.71:39829
server_dump_info_path | /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.29.219.72:46159
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.29.219.84:42611 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 20:09:18.657467 30573 log_verifier.cc:126] Checking tablet 09ac60ecdffa40b2a321c810d7890609
I20240501 20:09:18.667588 30573 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 09ac60ecdffa40b2a321c810d7890609
I20240501 20:09:18.667702 30573 log_verifier.cc:126] Checking tablet 0a9d19f112154a65b012072f9d0d4149
I20240501 20:09:18.677242 30573 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 0a9d19f112154a65b012072f9d0d4149
I20240501 20:09:18.677347 30573 log_verifier.cc:126] Checking tablet 1b76d484a0a7446081d3e7dde20af484
I20240501 20:09:18.689742 30573 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 1b76d484a0a7446081d3e7dde20af484
I20240501 20:09:18.689854 30573 log_verifier.cc:126] Checking tablet 1ef1cdbd7aa34985adb56e32c6eb64df
I20240501 20:09:18.695044 30573 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 1ef1cdbd7aa34985adb56e32c6eb64df
I20240501 20:09:18.695147 30573 log_verifier.cc:126] Checking tablet 2705dcf15dc54cbbb263fcfee0ee8aed
I20240501 20:09:18.705230 30573 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 2705dcf15dc54cbbb263fcfee0ee8aed
I20240501 20:09:18.705332 30573 log_verifier.cc:126] Checking tablet 2ad6ed0144eb4b619964ee4f86b9a842
I20240501 20:09:18.715240 30573 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 2ad6ed0144eb4b619964ee4f86b9a842
I20240501 20:09:18.715359 30573 log_verifier.cc:126] Checking tablet 3a1f02449b91442a897ac2278ee98e29
I20240501 20:09:18.725250 30573 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 3a1f02449b91442a897ac2278ee98e29
I20240501 20:09:18.725375 30573 log_verifier.cc:126] Checking tablet 5786820dd3d84b86a7a800b034a3e833
I20240501 20:09:18.734697 30573 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 5786820dd3d84b86a7a800b034a3e833
I20240501 20:09:18.734787 30573 log_verifier.cc:126] Checking tablet 5ae809988743421f8046716158b43a88
I20240501 20:09:18.744539 30573 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 5ae809988743421f8046716158b43a88
I20240501 20:09:18.744635 30573 log_verifier.cc:126] Checking tablet 6cee76b4a3ef4e7a87714c210cf705ac
I20240501 20:09:18.757012 30573 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 6cee76b4a3ef4e7a87714c210cf705ac
I20240501 20:09:18.757108 30573 log_verifier.cc:126] Checking tablet 76cd9ee2deb14a5aa9bb280b0a8dec53
I20240501 20:09:18.766597 30573 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 76cd9ee2deb14a5aa9bb280b0a8dec53
I20240501 20:09:18.766711 30573 log_verifier.cc:126] Checking tablet 998f81b7cc7e4df7832f4694c4f34731
I20240501 20:09:18.776912 30573 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 998f81b7cc7e4df7832f4694c4f34731
I20240501 20:09:18.777002 30573 log_verifier.cc:126] Checking tablet d2f958611c6c4f459055bd2a7cc75f15
I20240501 20:09:18.786343 30573 log_verifier.cc:177] Verified matching terms for 7 ops in tablet d2f958611c6c4f459055bd2a7cc75f15
I20240501 20:09:18.786439 30573 log_verifier.cc:126] Checking tablet e4b484eeb8424dbb9844f6b96fdacd34
I20240501 20:09:18.795948 30573 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e4b484eeb8424dbb9844f6b96fdacd34
I20240501 20:09:18.796031 30573 log_verifier.cc:126] Checking tablet e606598eee444113b75b26943d884754
I20240501 20:09:18.805649 30573 log_verifier.cc:177] Verified matching terms for 7 ops in tablet e606598eee444113b75b26943d884754
I20240501 20:09:18.805750 30573 log_verifier.cc:126] Checking tablet f71d46683ee74f0b8adb7ffdc4ec2688
I20240501 20:09:18.815575 30573 log_verifier.cc:177] Verified matching terms for 7 ops in tablet f71d46683ee74f0b8adb7ffdc4ec2688
I20240501 20:09:18.816625 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 543
I20240501 20:09:18.818055 666 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:18.984319 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 543
I20240501 20:09:19.011507 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 675
I20240501 20:09:19.012509 797 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:19.149027 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 675
I20240501 20:09:19.174516 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 1739
I20240501 20:09:19.186584 1862 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:19.300782 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 1739
I20240501 20:09:19.322137 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 1871
I20240501 20:09:19.328292 1993 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:19.442297 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 1871
I20240501 20:09:19.463641 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 2002
I20240501 20:09:19.469391 2123 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:19.583132 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 2002
I20240501 20:09:19.604373 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 2132
I20240501 20:09:19.610152 2254 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:19.724242 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 2132
I20240501 20:09:19.745756 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 2263
I20240501 20:09:19.751351 2386 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:19.864285 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 2263
I20240501 20:09:19.885636 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 3498
I20240501 20:09:19.891744 3619 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:19.994406 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 3498
I20240501 20:09:20.013298 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 453
I20240501 20:09:20.014317 514 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:20.162515 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 453
2024-05-01T20:09:20Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/1 (29496 ms)
[----------] 1 test from RebalanceParamTest (29496 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1
I20240501 20:09:20.232961 30573 test_util.cc:268] Using random seed: 265996778
I20240501 20:09:20.234898 30573 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:09:20.234966 30573 ts_itest-base.cc:113] --------------
I20240501 20:09:20.235040 30573 ts_itest-base.cc:114] 8 tablet servers
I20240501 20:09:20.235087 30573 ts_itest-base.cc:115] 3 replicas per TS
I20240501 20:09:20.235159 30573 ts_itest-base.cc:116] --------------
2024-05-01T20:09:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:09:20Z Disabled control of system clock
I20240501 20:09:20.267724 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.219.126:36407
--webserver_interface=127.29.219.126
--webserver_port=0
--builtin_ntp_servers=127.29.219.84:35419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.219.126:36407
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 20:09:20.594156 4061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:20.594487 4061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:20.594645 4061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:20.602259 4061 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:20.602391 4061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:09:20.602471 4061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:20.602556 4061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:09:20.602640 4061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:09:20.611757 4061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:35419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.29.219.126:36407
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.219.126:36407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.29.219.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:20.613128 4061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:20.614874 4061 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:20.628407 4067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:20.628463 4066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:20.629523 4069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:20.630523 4061 server_base.cc:1034] running on GCE node
I20240501 20:09:20.631544 4061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:20.633035 4061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:20.634258 4061 hybrid_clock.cc:648] HybridClock initialized: now 1714594160634202 us; error 51 us; skew 500 ppm
I20240501 20:09:20.634562 4061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:20.636844 4061 webserver.cc:460] Webserver started at http://127.29.219.126:38967/ using document root <none> and password file <none>
I20240501 20:09:20.637604 4061 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:20.637717 4061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:20.638021 4061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:20.640614 4061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ee64e9b209b3427cb57a00ccc1cd5617"
format_stamp: "Formatted at 2024-05-01 20:09:20 on dist-test-slave-h2cj"
I20240501 20:09:20.641360 4061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ee64e9b209b3427cb57a00ccc1cd5617"
format_stamp: "Formatted at 2024-05-01 20:09:20 on dist-test-slave-h2cj"
I20240501 20:09:20.646378 4061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240501 20:09:20.649686 4075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:20.650980 4061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20240501 20:09:20.651137 4061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ee64e9b209b3427cb57a00ccc1cd5617"
format_stamp: "Formatted at 2024-05-01 20:09:20 on dist-test-slave-h2cj"
I20240501 20:09:20.651325 4061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:20.669682 4061 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:20.670548 4061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:20.670855 4061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:20.688367 4061 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.126:36407
I20240501 20:09:20.688388 4127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.126:36407 every 8 connection(s)
I20240501 20:09:20.689757 4061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:09:20.693708 4128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:20.698801 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 4061
I20240501 20:09:20.698982 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:09:20.703856 4128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Bootstrap starting.
I20240501 20:09:20.708010 4128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:20.709161 4128 log.cc:824] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:20.711791 4128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: No bootstrap required, opened a new log
I20240501 20:09:20.719151 4128 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee64e9b209b3427cb57a00ccc1cd5617" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 36407 } }
I20240501 20:09:20.719485 4128 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:20.719578 4128 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee64e9b209b3427cb57a00ccc1cd5617, State: Initialized, Role: FOLLOWER
I20240501 20:09:20.720130 4128 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee64e9b209b3427cb57a00ccc1cd5617" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 36407 } }
I20240501 20:09:20.720326 4128 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:09:20.720440 4128 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:09:20.720580 4128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:20.722555 4128 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee64e9b209b3427cb57a00ccc1cd5617" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 36407 } }
I20240501 20:09:20.723047 4128 leader_election.cc:302] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee64e9b209b3427cb57a00ccc1cd5617; no voters:
I20240501 20:09:20.723567 4128 leader_election.cc:288] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:09:20.723701 4133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:20.725045 4133 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617 [term 1 LEADER]: Becoming Leader. State: Replica: ee64e9b209b3427cb57a00ccc1cd5617, State: Running, Role: LEADER
I20240501 20:09:20.725512 4128 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:09:20.725503 4133 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee64e9b209b3427cb57a00ccc1cd5617" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 36407 } }
I20240501 20:09:20.728210 4134 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee64e9b209b3427cb57a00ccc1cd5617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee64e9b209b3427cb57a00ccc1cd5617" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 36407 } } }
I20240501 20:09:20.728569 4134 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617 [sys.catalog]: This master's current role is: LEADER
I20240501 20:09:20.728942 4135 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee64e9b209b3427cb57a00ccc1cd5617. Latest consensus state: current_term: 1 leader_uuid: "ee64e9b209b3427cb57a00ccc1cd5617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee64e9b209b3427cb57a00ccc1cd5617" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 36407 } } }
I20240501 20:09:20.729178 4135 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617 [sys.catalog]: This master's current role is: LEADER
I20240501 20:09:20.729579 4142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:09:20.735925 4142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:09:20.744390 4142 catalog_manager.cc:1347] Generated new cluster ID: f76a124d7a8b4854936c832fd0263c8f
I20240501 20:09:20.744529 4142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:09:20.771448 4142 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:09:20.772673 4142 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:09:20.781047 4142 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Generated new TSK 0
I20240501 20:09:20.781898 4142 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:09:20.796571 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.65:0
--local_ip_for_outbound_sockets=127.29.219.65
--webserver_interface=127.29.219.65
--webserver_port=0
--tserver_master_addrs=127.29.219.126:36407
--builtin_ntp_servers=127.29.219.84:35419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:09:21.148020 4152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:21.148324 4152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:21.148415 4152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:21.156349 4152 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:21.156491 4152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:21.156656 4152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.65
I20240501 20:09:21.166265 4152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:35419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.29.219.65
--webserver_port=0
--tserver_master_addrs=127.29.219.126:36407
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.65
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:21.167654 4152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:21.169430 4152 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:21.181324 4157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:21.182294 4160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:21.183092 4158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:21.184204 4152 server_base.cc:1034] running on GCE node
I20240501 20:09:21.184852 4152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:21.185833 4152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:21.187023 4152 hybrid_clock.cc:648] HybridClock initialized: now 1714594161186978 us; error 43 us; skew 500 ppm
I20240501 20:09:21.187326 4152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:21.189536 4152 webserver.cc:460] Webserver started at http://127.29.219.65:45051/ using document root <none> and password file <none>
I20240501 20:09:21.190263 4152 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:21.190373 4152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:21.190701 4152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:21.195112 4152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "dca57823003e4729b769ca12367d7530"
format_stamp: "Formatted at 2024-05-01 20:09:21 on dist-test-slave-h2cj"
I20240501 20:09:21.196029 4152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "dca57823003e4729b769ca12367d7530"
format_stamp: "Formatted at 2024-05-01 20:09:21 on dist-test-slave-h2cj"
I20240501 20:09:21.201771 4152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 20:09:21.205374 4166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:21.206759 4152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:21.206898 4152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "dca57823003e4729b769ca12367d7530"
format_stamp: "Formatted at 2024-05-01 20:09:21 on dist-test-slave-h2cj"
I20240501 20:09:21.207068 4152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:21.223383 4152 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:21.224203 4152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:21.224433 4152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:21.225322 4152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:21.226871 4152 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:21.226986 4152 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:21.227074 4152 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:21.227128 4152 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:21.258342 4152 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.65:42941
I20240501 20:09:21.258389 4279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.65:42941 every 8 connection(s)
I20240501 20:09:21.259763 4152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:09:21.260845 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 4152
I20240501 20:09:21.261094 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:09:21.264971 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.66:0
--local_ip_for_outbound_sockets=127.29.219.66
--webserver_interface=127.29.219.66
--webserver_port=0
--tserver_master_addrs=127.29.219.126:36407
--builtin_ntp_servers=127.29.219.84:35419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:21.275069 4280 heartbeater.cc:347] Connected to a master server at 127.29.219.126:36407
I20240501 20:09:21.275365 4280 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:21.276162 4280 heartbeater.cc:511] Master 127.29.219.126:36407 requested a full tablet report, sending...
I20240501 20:09:21.278139 4093 ts_manager.cc:194] Registered new tserver with Master: dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:09:21.279599 4093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.65:53491
W20240501 20:09:21.594558 4284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:21.594939 4284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:21.595072 4284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:21.602777 4284 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:21.602913 4284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:21.603070 4284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.66
I20240501 20:09:21.612128 4284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:35419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.29.219.66
--webserver_port=0
--tserver_master_addrs=127.29.219.126:36407
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.66
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:21.613461 4284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:21.615243 4284 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:21.627693 4292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:21.628675 4290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:21.628978 4289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:21.630013 4284 server_base.cc:1034] running on GCE node
I20240501 20:09:21.630586 4284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:21.631565 4284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:21.632822 4284 hybrid_clock.cc:648] HybridClock initialized: now 1714594161632768 us; error 45 us; skew 500 ppm
I20240501 20:09:21.633133 4284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:21.635397 4284 webserver.cc:460] Webserver started at http://127.29.219.66:36875/ using document root <none> and password file <none>
I20240501 20:09:21.636134 4284 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:21.636246 4284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:21.636548 4284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:21.639271 4284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "10b47782558c41ee91f5e164a372049a"
format_stamp: "Formatted at 2024-05-01 20:09:21 on dist-test-slave-h2cj"
I20240501 20:09:21.640053 4284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "10b47782558c41ee91f5e164a372049a"
format_stamp: "Formatted at 2024-05-01 20:09:21 on dist-test-slave-h2cj"
I20240501 20:09:21.645291 4284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:09:21.648706 4298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:21.650022 4284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20240501 20:09:21.650174 4284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "10b47782558c41ee91f5e164a372049a"
format_stamp: "Formatted at 2024-05-01 20:09:21 on dist-test-slave-h2cj"
I20240501 20:09:21.650369 4284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:21.669070 4284 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:21.669914 4284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:21.670188 4284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:21.671147 4284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:21.672578 4284 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:21.672672 4284 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:21.672797 4284 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:21.672865 4284 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:21.704545 4284 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.66:40689
I20240501 20:09:21.704582 4412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.66:40689 every 8 connection(s)
I20240501 20:09:21.705873 4284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:09:21.707603 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 4284
I20240501 20:09:21.707738 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:09:21.710887 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.67:0
--local_ip_for_outbound_sockets=127.29.219.67
--webserver_interface=127.29.219.67
--webserver_port=0
--tserver_master_addrs=127.29.219.126:36407
--builtin_ntp_servers=127.29.219.84:35419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:21.718817 4413 heartbeater.cc:347] Connected to a master server at 127.29.219.126:36407
I20240501 20:09:21.719166 4413 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:21.720180 4413 heartbeater.cc:511] Master 127.29.219.126:36407 requested a full tablet report, sending...
I20240501 20:09:21.721788 4093 ts_manager.cc:194] Registered new tserver with Master: 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:21.722530 4093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.66:55257
W20240501 20:09:22.050621 4416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:22.050990 4416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:22.051149 4416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:22.059015 4416 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:22.059180 4416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:22.059365 4416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.67
I20240501 20:09:22.068950 4416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:35419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.29.219.67
--webserver_port=0
--tserver_master_addrs=127.29.219.126:36407
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.67
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:22.070360 4416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:22.072207 4416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:22.084401 4422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:22.085549 4425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:22.086567 4423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:22.087452 4416 server_base.cc:1034] running on GCE node
I20240501 20:09:22.088073 4416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:22.089015 4416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:22.090202 4416 hybrid_clock.cc:648] HybridClock initialized: now 1714594162090152 us; error 42 us; skew 500 ppm
I20240501 20:09:22.090504 4416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:22.092730 4416 webserver.cc:460] Webserver started at http://127.29.219.67:35583/ using document root <none> and password file <none>
I20240501 20:09:22.093465 4416 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:22.093575 4416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:22.093864 4416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:22.096599 4416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "3e974c03d3c546f9ab6c91e762a5426b"
format_stamp: "Formatted at 2024-05-01 20:09:22 on dist-test-slave-h2cj"
I20240501 20:09:22.097379 4416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "3e974c03d3c546f9ab6c91e762a5426b"
format_stamp: "Formatted at 2024-05-01 20:09:22 on dist-test-slave-h2cj"
I20240501 20:09:22.102486 4416 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 20:09:22.105820 4431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:22.107115 4416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:22.107278 4416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3e974c03d3c546f9ab6c91e762a5426b"
format_stamp: "Formatted at 2024-05-01 20:09:22 on dist-test-slave-h2cj"
I20240501 20:09:22.107461 4416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:22.134965 4416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:22.135821 4416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:22.136112 4416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:22.137096 4416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:22.138657 4416 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:22.138751 4416 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:22.138864 4416 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:22.138934 4416 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:22.170076 4416 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.67:34595
I20240501 20:09:22.170150 4543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.67:34595 every 8 connection(s)
I20240501 20:09:22.171447 4416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:09:22.178047 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 4416
I20240501 20:09:22.178236 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:09:22.181433 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.68:0
--local_ip_for_outbound_sockets=127.29.219.68
--webserver_interface=127.29.219.68
--webserver_port=0
--tserver_master_addrs=127.29.219.126:36407
--builtin_ntp_servers=127.29.219.84:35419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:22.183782 4544 heartbeater.cc:347] Connected to a master server at 127.29.219.126:36407
I20240501 20:09:22.184154 4544 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:22.185213 4544 heartbeater.cc:511] Master 127.29.219.126:36407 requested a full tablet report, sending...
I20240501 20:09:22.186846 4093 ts_manager.cc:194] Registered new tserver with Master: 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:09:22.187600 4093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.67:39233
I20240501 20:09:22.282433 4280 heartbeater.cc:503] Master 127.29.219.126:36407 was elected leader, sending a full tablet report...
W20240501 20:09:22.516475 4547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:22.516795 4547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:22.516928 4547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:22.524250 4547 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:22.524381 4547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:22.524530 4547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.68
I20240501 20:09:22.533552 4547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:35419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.29.219.68
--webserver_port=0
--tserver_master_addrs=127.29.219.126:36407
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.68
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:22.534953 4547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:22.536718 4547 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:22.547915 4552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:22.548943 4555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:22.549948 4553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:22.550864 4547 server_base.cc:1034] running on GCE node
I20240501 20:09:22.551405 4547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:22.552327 4547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:22.553551 4547 hybrid_clock.cc:648] HybridClock initialized: now 1714594162553497 us; error 34 us; skew 500 ppm
I20240501 20:09:22.553851 4547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:22.556120 4547 webserver.cc:460] Webserver started at http://127.29.219.68:35093/ using document root <none> and password file <none>
I20240501 20:09:22.556854 4547 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:22.556962 4547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:22.557256 4547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:22.559813 4547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
format_stamp: "Formatted at 2024-05-01 20:09:22 on dist-test-slave-h2cj"
I20240501 20:09:22.560539 4547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
format_stamp: "Formatted at 2024-05-01 20:09:22 on dist-test-slave-h2cj"
I20240501 20:09:22.565564 4547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:09:22.568753 4561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:22.570076 4547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:22.570251 4547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
format_stamp: "Formatted at 2024-05-01 20:09:22 on dist-test-slave-h2cj"
I20240501 20:09:22.570438 4547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:22.587638 4547 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:22.588455 4547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:22.588713 4547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:22.589629 4547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:22.591125 4547 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:22.591230 4547 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:22.591337 4547 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:22.591406 4547 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:22.622714 4547 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.68:45205
I20240501 20:09:22.622752 4675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.68:45205 every 8 connection(s)
I20240501 20:09:22.624210 4547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:09:22.633522 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 4547
I20240501 20:09:22.633706 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:09:22.636615 4676 heartbeater.cc:347] Connected to a master server at 127.29.219.126:36407
I20240501 20:09:22.636909 4676 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:22.637228 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.69:0
--local_ip_for_outbound_sockets=127.29.219.69
--webserver_interface=127.29.219.69
--webserver_port=0
--tserver_master_addrs=127.29.219.126:36407
--builtin_ntp_servers=127.29.219.84:35419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:22.637769 4676 heartbeater.cc:511] Master 127.29.219.126:36407 requested a full tablet report, sending...
I20240501 20:09:22.639148 4093 ts_manager.cc:194] Registered new tserver with Master: 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:22.639914 4093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.68:47369
I20240501 20:09:22.725504 4413 heartbeater.cc:503] Master 127.29.219.126:36407 was elected leader, sending a full tablet report...
W20240501 20:09:22.962363 4679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:22.962698 4679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:22.962836 4679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:22.970209 4679 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:22.970360 4679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:22.970521 4679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.69
I20240501 20:09:22.979701 4679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:35419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.29.219.69
--webserver_port=0
--tserver_master_addrs=127.29.219.126:36407
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.69
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:22.981084 4679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:22.982852 4679 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:22.995225 4685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:22.996206 4687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:22.997241 4684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:22.998225 4679 server_base.cc:1034] running on GCE node
I20240501 20:09:22.998858 4679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:22.999902 4679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:23.001111 4679 hybrid_clock.cc:648] HybridClock initialized: now 1714594163001044 us; error 70 us; skew 500 ppm
I20240501 20:09:23.001428 4679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:23.003666 4679 webserver.cc:460] Webserver started at http://127.29.219.69:43169/ using document root <none> and password file <none>
I20240501 20:09:23.004441 4679 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:23.004551 4679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:23.004863 4679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:23.007632 4679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-h2cj"
I20240501 20:09:23.008409 4679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-h2cj"
I20240501 20:09:23.013823 4679 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20240501 20:09:23.017262 4694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:23.018695 4679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240501 20:09:23.018844 4679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-h2cj"
I20240501 20:09:23.019045 4679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:23.040992 4679 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:23.041853 4679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:23.042109 4679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:23.043071 4679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:23.044484 4679 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:23.044570 4679 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:23.044677 4679 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:23.044744 4679 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:23.075322 4679 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.69:46237
I20240501 20:09:23.075376 4806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.69:46237 every 8 connection(s)
I20240501 20:09:23.076624 4679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:09:23.077494 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 4679
I20240501 20:09:23.077697 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 20:09:23.081044 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.70:0
--local_ip_for_outbound_sockets=127.29.219.70
--webserver_interface=127.29.219.70
--webserver_port=0
--tserver_master_addrs=127.29.219.126:36407
--builtin_ntp_servers=127.29.219.84:35419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:23.090152 4807 heartbeater.cc:347] Connected to a master server at 127.29.219.126:36407
I20240501 20:09:23.090507 4807 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:23.091626 4807 heartbeater.cc:511] Master 127.29.219.126:36407 requested a full tablet report, sending...
I20240501 20:09:23.093145 4093 ts_manager.cc:194] Registered new tserver with Master: ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:09:23.093904 4093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.69:48019
I20240501 20:09:23.190208 4544 heartbeater.cc:503] Master 127.29.219.126:36407 was elected leader, sending a full tablet report...
W20240501 20:09:23.420498 4810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:23.420832 4810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:23.420976 4810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:23.429117 4810 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:23.429261 4810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:23.429417 4810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.70
I20240501 20:09:23.438656 4810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:35419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.29.219.70
--webserver_port=0
--tserver_master_addrs=127.29.219.126:36407
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.70
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:23.440024 4810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:23.441804 4810 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:23.454197 4815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:23.455281 4816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:23.456288 4818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:23.457134 4810 server_base.cc:1034] running on GCE node
I20240501 20:09:23.457796 4810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:23.458812 4810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:23.460036 4810 hybrid_clock.cc:648] HybridClock initialized: now 1714594163459977 us; error 54 us; skew 500 ppm
I20240501 20:09:23.460367 4810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:23.462754 4810 webserver.cc:460] Webserver started at http://127.29.219.70:39459/ using document root <none> and password file <none>
I20240501 20:09:23.463493 4810 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:23.463609 4810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:23.463968 4810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:23.466679 4810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "db0742ee3860468590eefef6ae6074fd"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-h2cj"
I20240501 20:09:23.467485 4810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "db0742ee3860468590eefef6ae6074fd"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-h2cj"
I20240501 20:09:23.472842 4810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20240501 20:09:23.476452 4824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:23.477931 4810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 20:09:23.478176 4810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "db0742ee3860468590eefef6ae6074fd"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-h2cj"
I20240501 20:09:23.478431 4810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:23.503858 4810 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:23.504814 4810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:23.505095 4810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:23.506047 4810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:23.507659 4810 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:23.507771 4810 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:23.507887 4810 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:23.507958 4810 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:23.539901 4810 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.70:36321
I20240501 20:09:23.539945 4940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.70:36321 every 8 connection(s)
I20240501 20:09:23.541245 4810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:09:23.545634 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 4810
I20240501 20:09:23.545809 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 20:09:23.548951 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.71:0
--local_ip_for_outbound_sockets=127.29.219.71
--webserver_interface=127.29.219.71
--webserver_port=0
--tserver_master_addrs=127.29.219.126:36407
--builtin_ntp_servers=127.29.219.84:35419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:23.554381 4941 heartbeater.cc:347] Connected to a master server at 127.29.219.126:36407
I20240501 20:09:23.554791 4941 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:23.555682 4941 heartbeater.cc:511] Master 127.29.219.126:36407 requested a full tablet report, sending...
I20240501 20:09:23.557108 4093 ts_manager.cc:194] Registered new tserver with Master: db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:09:23.557870 4093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.70:58753
I20240501 20:09:23.642482 4676 heartbeater.cc:503] Master 127.29.219.126:36407 was elected leader, sending a full tablet report...
W20240501 20:09:23.891688 4944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:23.892009 4944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:23.892144 4944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:23.904440 4944 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:23.904615 4944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:23.904877 4944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.71
I20240501 20:09:23.920780 4944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:35419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.29.219.71
--webserver_port=0
--tserver_master_addrs=127.29.219.126:36407
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.71
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:23.922150 4944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:23.923956 4944 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:23.936224 4949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:23.937225 4952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:23.938067 4950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:23.938925 4944 server_base.cc:1034] running on GCE node
I20240501 20:09:23.939541 4944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:23.940542 4944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:23.941772 4944 hybrid_clock.cc:648] HybridClock initialized: now 1714594163941728 us; error 38 us; skew 500 ppm
I20240501 20:09:23.942098 4944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:23.944347 4944 webserver.cc:460] Webserver started at http://127.29.219.71:33093/ using document root <none> and password file <none>
I20240501 20:09:23.945087 4944 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:23.945199 4944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:23.945489 4944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:23.948141 4944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "bceeae1c79964acabdae8ff771bf0c97"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-h2cj"
I20240501 20:09:23.948913 4944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "bceeae1c79964acabdae8ff771bf0c97"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-h2cj"
I20240501 20:09:23.954082 4944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240501 20:09:23.957660 4958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:23.959169 4944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:23.959333 4944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "bceeae1c79964acabdae8ff771bf0c97"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-h2cj"
I20240501 20:09:23.959537 4944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:23.986169 4944 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:23.987115 4944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:23.987390 4944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:23.988379 4944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:23.989892 4944 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:23.989993 4944 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:23.990113 4944 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:23.990185 4944 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:24.022899 4944 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.71:39447
I20240501 20:09:24.022989 5071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.71:39447 every 8 connection(s)
I20240501 20:09:24.024363 4944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 20:09:24.032322 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 4944
I20240501 20:09:24.032522 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 20:09:24.035738 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.72:0
--local_ip_for_outbound_sockets=127.29.219.72
--webserver_interface=127.29.219.72
--webserver_port=0
--tserver_master_addrs=127.29.219.126:36407
--builtin_ntp_servers=127.29.219.84:35419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:24.038231 5072 heartbeater.cc:347] Connected to a master server at 127.29.219.126:36407
I20240501 20:09:24.038559 5072 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:24.039479 5072 heartbeater.cc:511] Master 127.29.219.126:36407 requested a full tablet report, sending...
I20240501 20:09:24.041090 4093 ts_manager.cc:194] Registered new tserver with Master: bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:09:24.042258 4093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.71:33489
I20240501 20:09:24.097105 4807 heartbeater.cc:503] Master 127.29.219.126:36407 was elected leader, sending a full tablet report...
W20240501 20:09:24.382319 5075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:24.382644 5075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:24.382743 5075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:24.390424 5075 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:24.390555 5075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:24.390722 5075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.72
I20240501 20:09:24.399991 5075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:35419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.29.219.72
--webserver_port=0
--tserver_master_addrs=127.29.219.126:36407
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.72
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:24.401296 5075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:24.403066 5075 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:24.415241 5081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:24.416137 5080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:24.416944 5083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:24.417906 5075 server_base.cc:1034] running on GCE node
I20240501 20:09:24.418566 5075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:24.419618 5075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:24.420847 5075 hybrid_clock.cc:648] HybridClock initialized: now 1714594164420786 us; error 59 us; skew 500 ppm
I20240501 20:09:24.421182 5075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:24.429281 5075 webserver.cc:460] Webserver started at http://127.29.219.72:44747/ using document root <none> and password file <none>
I20240501 20:09:24.430084 5075 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:24.430193 5075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:24.430502 5075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:09:24.433163 5075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "b9957d4086bc4a76b18ed9265a5739d5"
format_stamp: "Formatted at 2024-05-01 20:09:24 on dist-test-slave-h2cj"
I20240501 20:09:24.433939 5075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "b9957d4086bc4a76b18ed9265a5739d5"
format_stamp: "Formatted at 2024-05-01 20:09:24 on dist-test-slave-h2cj"
I20240501 20:09:24.439324 5075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 20:09:24.442601 5089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:24.443950 5075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:09:24.444106 5075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "b9957d4086bc4a76b18ed9265a5739d5"
format_stamp: "Formatted at 2024-05-01 20:09:24 on dist-test-slave-h2cj"
I20240501 20:09:24.444289 5075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:24.465057 5075 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:24.465910 5075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:24.466181 5075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:24.467154 5075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:24.468600 5075 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:24.468688 5075 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:24.468807 5075 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:24.468871 5075 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:24.499133 5075 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.72:39817
I20240501 20:09:24.499181 5202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.72:39817 every 8 connection(s)
I20240501 20:09:24.500422 5075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 20:09:24.508813 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 5075
I20240501 20:09:24.508986 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 20:09:24.512674 5203 heartbeater.cc:347] Connected to a master server at 127.29.219.126:36407
I20240501 20:09:24.512962 5203 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:24.513729 5203 heartbeater.cc:511] Master 127.29.219.126:36407 requested a full tablet report, sending...
I20240501 20:09:24.514987 4093 ts_manager.cc:194] Registered new tserver with Master: b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:09:24.515723 4093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.72:34895
I20240501 20:09:24.525305 30573 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 20:09:24.543081 30573 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 20:09:24.543243 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 4679
I20240501 20:09:24.551630 4802 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:24.560631 4941 heartbeater.cc:503] Master 127.29.219.126:36407 was elected leader, sending a full tablet report...
I20240501 20:09:24.634344 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 4679
I20240501 20:09:24.650908 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 4810
I20240501 20:09:24.658731 4936 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:24.738777 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 4810
I20240501 20:09:24.755484 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 4944
I20240501 20:09:24.763603 5067 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:24.846856 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 4944
I20240501 20:09:24.864786 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 5075
I20240501 20:09:24.876441 5198 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:09:24.965971 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 5075
I20240501 20:09:28.735853 4093 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:28.790352 4611 tablet_service.cc:1467] Processing CreateTablet for tablet c7e9b168477a4dce9c70ed20468711d9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c8faa922e6eb4703821285891838ebab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:28.791622 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 8ddee2fd0dfd4e3295cbfad7edd5add6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c8faa922e6eb4703821285891838ebab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:28.793026 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ddee2fd0dfd4e3295cbfad7edd5add6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:28.795521 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7e9b168477a4dce9c70ed20468711d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:28.795073 4347 tablet_service.cc:1467] Processing CreateTablet for tablet bfd49691236341dea4af4ead4450286e (DEFAULT_TABLE table=rebalance_test_table_0 [id=c8faa922e6eb4703821285891838ebab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:28.796797 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd49691236341dea4af4ead4450286e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:28.799192 4346 tablet_service.cc:1467] Processing CreateTablet for tablet 8ddee2fd0dfd4e3295cbfad7edd5add6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c8faa922e6eb4703821285891838ebab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:28.801288 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ddee2fd0dfd4e3295cbfad7edd5add6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:28.801759 4214 tablet_service.cc:1467] Processing CreateTablet for tablet 8ddee2fd0dfd4e3295cbfad7edd5add6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c8faa922e6eb4703821285891838ebab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:28.803496 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ddee2fd0dfd4e3295cbfad7edd5add6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:28.803833 4215 tablet_service.cc:1467] Processing CreateTablet for tablet bfd49691236341dea4af4ead4450286e (DEFAULT_TABLE table=rebalance_test_table_0 [id=c8faa922e6eb4703821285891838ebab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:28.804415 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd49691236341dea4af4ead4450286e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:28.806389 4478 tablet_service.cc:1467] Processing CreateTablet for tablet bfd49691236341dea4af4ead4450286e (DEFAULT_TABLE table=rebalance_test_table_0 [id=c8faa922e6eb4703821285891838ebab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:28.808229 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd49691236341dea4af4ead4450286e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:28.793404 4348 tablet_service.cc:1467] Processing CreateTablet for tablet c7e9b168477a4dce9c70ed20468711d9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c8faa922e6eb4703821285891838ebab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:28.806370 4479 tablet_service.cc:1467] Processing CreateTablet for tablet c7e9b168477a4dce9c70ed20468711d9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c8faa922e6eb4703821285891838ebab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:28.811259 4348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7e9b168477a4dce9c70ed20468711d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:28.811519 4479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7e9b168477a4dce9c70ed20468711d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:28.839711 5233 tablet_bootstrap.cc:492] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:28.839711 5234 tablet_bootstrap.cc:492] T bfd49691236341dea4af4ead4450286e P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:28.844189 5234 tablet_bootstrap.cc:654] T bfd49691236341dea4af4ead4450286e P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:28.844189 5233 tablet_bootstrap.cc:654] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:28.846340 5232 tablet_bootstrap.cc:492] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:28.846449 5233 log.cc:824] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:28.847851 5235 tablet_bootstrap.cc:492] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:28.851629 5234 log.cc:824] T bfd49691236341dea4af4ead4450286e P 10b47782558c41ee91f5e164a372049a: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:28.851964 5235 tablet_bootstrap.cc:654] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:28.852243 5232 tablet_bootstrap.cc:654] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:28.853685 5235 log.cc:824] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:28.854080 5232 log.cc:824] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:28.858736 5233 tablet_bootstrap.cc:492] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:28.859171 5233 ts_tablet_manager.cc:1367] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.003s
I20240501 20:09:28.860594 5232 tablet_bootstrap.cc:492] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:28.860966 5232 ts_tablet_manager.cc:1367] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.000s
I20240501 20:09:28.869591 5233 raft_consensus.cc:348] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:28.870304 5233 raft_consensus.cc:374] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:28.870481 5233 raft_consensus.cc:729] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:28.872521 5232 raft_consensus.cc:348] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:28.873147 5235 tablet_bootstrap.cc:492] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:28.873139 5232 raft_consensus.cc:374] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:28.872987 5233 consensus_queue.cc:260] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:28.873260 5232 raft_consensus.cc:729] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:28.873528 5234 tablet_bootstrap.cc:492] T bfd49691236341dea4af4ead4450286e P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:28.873579 5235 ts_tablet_manager.cc:1367] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.008s
I20240501 20:09:28.874001 5234 ts_tablet_manager.cc:1367] T bfd49691236341dea4af4ead4450286e P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.020s
I20240501 20:09:28.874106 5232 consensus_queue.cc:260] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:28.882050 5233 ts_tablet_manager.cc:1398] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.023s user 0.008s sys 0.010s
I20240501 20:09:28.885479 5235 raft_consensus.cc:348] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:28.886181 5235 raft_consensus.cc:374] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:28.886382 5235 raft_consensus.cc:729] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:28.887462 5235 consensus_queue.cc:260] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:28.886981 5234 raft_consensus.cc:348] T bfd49691236341dea4af4ead4450286e P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:28.888250 5234 raft_consensus.cc:374] T bfd49691236341dea4af4ead4450286e P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:28.888530 5234 raft_consensus.cc:729] T bfd49691236341dea4af4ead4450286e P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:28.889719 5233 tablet_bootstrap.cc:492] T bfd49691236341dea4af4ead4450286e P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:28.890244 5234 consensus_queue.cc:260] T bfd49691236341dea4af4ead4450286e P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:28.892552 5233 tablet_bootstrap.cc:654] T bfd49691236341dea4af4ead4450286e P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:28.894644 5232 ts_tablet_manager.cc:1398] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.033s user 0.014s sys 0.014s
I20240501 20:09:28.895258 5232 tablet_bootstrap.cc:492] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:28.896026 5235 ts_tablet_manager.cc:1398] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.022s user 0.016s sys 0.004s
I20240501 20:09:28.896744 5235 tablet_bootstrap.cc:492] T c7e9b168477a4dce9c70ed20468711d9 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:28.899300 5235 tablet_bootstrap.cc:654] T c7e9b168477a4dce9c70ed20468711d9 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:28.903208 5232 tablet_bootstrap.cc:654] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:28.905567 5234 ts_tablet_manager.cc:1398] T bfd49691236341dea4af4ead4450286e P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.031s user 0.023s sys 0.000s
I20240501 20:09:28.908390 5234 tablet_bootstrap.cc:492] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:28.910984 5235 tablet_bootstrap.cc:492] T c7e9b168477a4dce9c70ed20468711d9 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:28.911379 5235 ts_tablet_manager.cc:1367] T c7e9b168477a4dce9c70ed20468711d9 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20240501 20:09:28.912963 5234 tablet_bootstrap.cc:654] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:28.913290 5235 raft_consensus.cc:348] T c7e9b168477a4dce9c70ed20468711d9 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:28.913589 5235 raft_consensus.cc:374] T c7e9b168477a4dce9c70ed20468711d9 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:28.913712 5235 raft_consensus.cc:729] T c7e9b168477a4dce9c70ed20468711d9 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:28.914183 5235 consensus_queue.cc:260] T c7e9b168477a4dce9c70ed20468711d9 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:28.915472 5235 ts_tablet_manager.cc:1398] T c7e9b168477a4dce9c70ed20468711d9 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 20:09:28.917529 5232 tablet_bootstrap.cc:492] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:28.917763 5232 ts_tablet_manager.cc:1367] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.008s
I20240501 20:09:28.919096 5232 raft_consensus.cc:348] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:28.919373 5232 raft_consensus.cc:374] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:28.919505 5232 raft_consensus.cc:729] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:28.919364 5233 tablet_bootstrap.cc:492] T bfd49691236341dea4af4ead4450286e P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:28.919754 5233 ts_tablet_manager.cc:1367] T bfd49691236341dea4af4ead4450286e P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.005s
I20240501 20:09:28.919775 5232 consensus_queue.cc:260] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:28.919812 5234 tablet_bootstrap.cc:492] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:28.920497 5232 ts_tablet_manager.cc:1398] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240501 20:09:28.920964 5233 raft_consensus.cc:348] T bfd49691236341dea4af4ead4450286e P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:28.921200 5233 raft_consensus.cc:374] T bfd49691236341dea4af4ead4450286e P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:28.921304 5233 raft_consensus.cc:729] T bfd49691236341dea4af4ead4450286e P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:28.921643 5234 ts_tablet_manager.cc:1367] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.001s
I20240501 20:09:28.921689 5233 consensus_queue.cc:260] T bfd49691236341dea4af4ead4450286e P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:28.922287 5233 ts_tablet_manager.cc:1398] T bfd49691236341dea4af4ead4450286e P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:28.923106 5234 raft_consensus.cc:348] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:28.923424 5234 raft_consensus.cc:374] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:28.923600 5234 raft_consensus.cc:729] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:28.923918 5234 consensus_queue.cc:260] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:28.924619 5234 ts_tablet_manager.cc:1398] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:28.925022 5234 tablet_bootstrap.cc:492] T c7e9b168477a4dce9c70ed20468711d9 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:28.927138 5234 tablet_bootstrap.cc:654] T c7e9b168477a4dce9c70ed20468711d9 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
W20240501 20:09:28.928673 4545 tablet.cc:2369] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:28.932536 5234 tablet_bootstrap.cc:492] T c7e9b168477a4dce9c70ed20468711d9 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:28.932799 5234 ts_tablet_manager.cc:1367] T c7e9b168477a4dce9c70ed20468711d9 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240501 20:09:28.934055 5234 raft_consensus.cc:348] T c7e9b168477a4dce9c70ed20468711d9 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:28.934420 5234 raft_consensus.cc:374] T c7e9b168477a4dce9c70ed20468711d9 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:28.934502 5234 raft_consensus.cc:729] T c7e9b168477a4dce9c70ed20468711d9 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:28.934953 5234 consensus_queue.cc:260] T c7e9b168477a4dce9c70ed20468711d9 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:28.935730 5234 ts_tablet_manager.cc:1398] T c7e9b168477a4dce9c70ed20468711d9 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:28.943079 5240 raft_consensus.cc:482] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:28.943413 5240 raft_consensus.cc:504] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:28.945973 5240 leader_election.cc:288] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:28.950313 5241 raft_consensus.cc:482] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:28.950678 5241 raft_consensus.cc:504] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:28.953159 5241 leader_election.cc:288] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:09:28.958330 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ddee2fd0dfd4e3295cbfad7edd5add6" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:28.959026 4368 raft_consensus.cc:2454] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:28.959939 4169 leader_election.cc:302] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:28.961470 5240 raft_consensus.cc:2781] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:28.961697 5240 raft_consensus.cc:482] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:28.961467 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ddee2fd0dfd4e3295cbfad7edd5add6" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:28.961818 5240 raft_consensus.cc:3037] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:28.962097 4631 raft_consensus.cc:2454] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
W20240501 20:09:28.963003 4414 tablet.cc:2369] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 10b47782558c41ee91f5e164a372049a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:28.965186 5240 raft_consensus.cc:504] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:28.966176 5240 leader_election.cc:288] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:28.966599 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ddee2fd0dfd4e3295cbfad7edd5add6" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:28.966938 4631 raft_consensus.cc:3037] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:28.967402 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ddee2fd0dfd4e3295cbfad7edd5add6" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:28.967651 4368 raft_consensus.cc:3037] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:28.970846 4631 raft_consensus.cc:2454] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:28.971091 4368 raft_consensus.cc:2454] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:28.970961 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e9b168477a4dce9c70ed20468711d9" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:28.971577 4170 leader_election.cc:302] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:28.971882 4499 raft_consensus.cc:2454] T c7e9b168477a4dce9c70ed20468711d9 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:09:28.972337 5240 raft_consensus.cc:2781] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:28.972847 4564 leader_election.cc:302] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:28.973275 5240 raft_consensus.cc:686] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:09:28.973409 5241 raft_consensus.cc:2781] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:28.973570 5241 raft_consensus.cc:482] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:28.973687 5241 raft_consensus.cc:3037] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:28.974073 5240 consensus_queue.cc:237] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:28.975849 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e9b168477a4dce9c70ed20468711d9" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:28.976143 4368 raft_consensus.cc:2454] T c7e9b168477a4dce9c70ed20468711d9 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:09:28.977416 5241 raft_consensus.cc:504] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:28.978266 5241 leader_election.cc:288] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:09:28.981834 4092 catalog_manager.cc:5484] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:28.983237 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e9b168477a4dce9c70ed20468711d9" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:28.983501 4368 raft_consensus.cc:3037] T c7e9b168477a4dce9c70ed20468711d9 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:28.986673 4368 raft_consensus.cc:2454] T c7e9b168477a4dce9c70ed20468711d9 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:09:28.987201 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7e9b168477a4dce9c70ed20468711d9" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:28.987449 4499 raft_consensus.cc:3037] T c7e9b168477a4dce9c70ed20468711d9 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:28.989957 4499 raft_consensus.cc:2454] T c7e9b168477a4dce9c70ed20468711d9 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:09:28.990548 4564 leader_election.cc:302] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 1d34e18a1d394d3f91c9f50d1c7bea4e; no voters:
I20240501 20:09:28.991114 5241 raft_consensus.cc:2781] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:28.991734 5241 raft_consensus.cc:686] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Becoming Leader. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Running, Role: LEADER
I20240501 20:09:28.992547 5241 consensus_queue.cc:237] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:28.997977 4092 catalog_manager.cc:5484] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68). New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:29.007887 5242 raft_consensus.cc:482] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:29.008185 5242 raft_consensus.cc:504] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:29.010864 5242 leader_election.cc:288] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
W20240501 20:09:29.018126 4281 tablet.cc:2369] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:29.023591 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd49691236341dea4af4ead4450286e" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:29.024060 4368 raft_consensus.cc:2454] T bfd49691236341dea4af4ead4450286e P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:29.025286 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd49691236341dea4af4ead4450286e" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:29.025547 4434 leader_election.cc:302] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:29.025785 4235 raft_consensus.cc:2454] T bfd49691236341dea4af4ead4450286e P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:29.026103 5242 raft_consensus.cc:2781] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:29.026263 5242 raft_consensus.cc:482] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:29.026371 5242 raft_consensus.cc:3037] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:29.030239 5242 raft_consensus.cc:504] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:29.031232 5242 leader_election.cc:288] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Requested vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:29.031853 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd49691236341dea4af4ead4450286e" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:29.032130 4235 raft_consensus.cc:3037] T bfd49691236341dea4af4ead4450286e P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:29.032088 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd49691236341dea4af4ead4450286e" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:29.032280 4368 raft_consensus.cc:3037] T bfd49691236341dea4af4ead4450286e P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:29.034360 4368 raft_consensus.cc:2454] T bfd49691236341dea4af4ead4450286e P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:29.034833 4434 leader_election.cc:302] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:29.035162 5242 raft_consensus.cc:2781] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:29.035692 5242 raft_consensus.cc:686] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:09:29.035866 4235 raft_consensus.cc:2454] T bfd49691236341dea4af4ead4450286e P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:29.036289 5242 consensus_queue.cc:237] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:29.043082 4092 catalog_manager.cc:5484] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } }
W20240501 20:09:29.131289 4677 tablet.cc:2369] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:29.386667 5241 consensus_queue.cc:1035] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:29.394860 5267 consensus_queue.cc:1035] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:29.416715 5240 consensus_queue.cc:1035] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:29.424697 5275 consensus_queue.cc:1035] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:29.428709 5263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:29.429050 5263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:29.437517 5263 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:09:29.437546 5242 consensus_queue.cc:1035] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:29.446815 5242 consensus_queue.cc:1035] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 257.174 ms
time per row: 0.128587 ms
I20240501 20:09:29.851579 4087 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:29.862440 4214 tablet_service.cc:1467] Processing CreateTablet for tablet 7725fa9f98e9439797a532ed6f2f7dba (DEFAULT_TABLE table=rebalance_test_table_1 [id=eccfbde58fa5455b9f89a20ad3a37f39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:29.862439 4479 tablet_service.cc:1467] Processing CreateTablet for tablet 7725fa9f98e9439797a532ed6f2f7dba (DEFAULT_TABLE table=rebalance_test_table_1 [id=eccfbde58fa5455b9f89a20ad3a37f39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:29.862982 4479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7725fa9f98e9439797a532ed6f2f7dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:29.862972 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7725fa9f98e9439797a532ed6f2f7dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:29.863440 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 7725fa9f98e9439797a532ed6f2f7dba (DEFAULT_TABLE table=rebalance_test_table_1 [id=eccfbde58fa5455b9f89a20ad3a37f39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:29.863852 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7725fa9f98e9439797a532ed6f2f7dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:29.865134 4478 tablet_service.cc:1467] Processing CreateTablet for tablet fcff4be357084bf0903cfb1c0f62a866 (DEFAULT_TABLE table=rebalance_test_table_1 [id=eccfbde58fa5455b9f89a20ad3a37f39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:29.865523 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcff4be357084bf0903cfb1c0f62a866. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:29.867344 4611 tablet_service.cc:1467] Processing CreateTablet for tablet fcff4be357084bf0903cfb1c0f62a866 (DEFAULT_TABLE table=rebalance_test_table_1 [id=eccfbde58fa5455b9f89a20ad3a37f39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:29.867522 4477 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf6bcf89d444b339612d731af604d0a (DEFAULT_TABLE table=rebalance_test_table_1 [id=eccfbde58fa5455b9f89a20ad3a37f39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:29.867750 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcff4be357084bf0903cfb1c0f62a866. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:29.868005 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf6bcf89d444b339612d731af604d0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:29.874173 4347 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf6bcf89d444b339612d731af604d0a (DEFAULT_TABLE table=rebalance_test_table_1 [id=eccfbde58fa5455b9f89a20ad3a37f39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:29.874910 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf6bcf89d444b339612d731af604d0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:29.879832 4215 tablet_service.cc:1467] Processing CreateTablet for tablet fcff4be357084bf0903cfb1c0f62a866 (DEFAULT_TABLE table=rebalance_test_table_1 [id=eccfbde58fa5455b9f89a20ad3a37f39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:29.879947 4213 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf6bcf89d444b339612d731af604d0a (DEFAULT_TABLE table=rebalance_test_table_1 [id=eccfbde58fa5455b9f89a20ad3a37f39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:29.880630 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcff4be357084bf0903cfb1c0f62a866. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:29.881421 5331 tablet_bootstrap.cc:492] T fcff4be357084bf0903cfb1c0f62a866 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:29.881945 5332 tablet_bootstrap.cc:492] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:29.883076 4213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf6bcf89d444b339612d731af604d0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:29.883988 5332 tablet_bootstrap.cc:654] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:29.885453 5331 tablet_bootstrap.cc:654] T fcff4be357084bf0903cfb1c0f62a866 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:29.890363 5335 tablet_bootstrap.cc:492] T fcff4be357084bf0903cfb1c0f62a866 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:29.898030 5335 tablet_bootstrap.cc:654] T fcff4be357084bf0903cfb1c0f62a866 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:29.901727 5332 tablet_bootstrap.cc:492] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:29.901937 5332 ts_tablet_manager.cc:1367] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.004s
I20240501 20:09:29.902307 5331 tablet_bootstrap.cc:492] T fcff4be357084bf0903cfb1c0f62a866 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:29.902464 5331 ts_tablet_manager.cc:1367] T fcff4be357084bf0903cfb1c0f62a866 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.000s
I20240501 20:09:29.902997 5332 raft_consensus.cc:348] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:29.903262 5332 raft_consensus.cc:374] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:29.903393 5332 raft_consensus.cc:729] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:29.903470 5331 raft_consensus.cc:348] T fcff4be357084bf0903cfb1c0f62a866 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:29.903717 5331 raft_consensus.cc:374] T fcff4be357084bf0903cfb1c0f62a866 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:29.903825 5331 raft_consensus.cc:729] T fcff4be357084bf0903cfb1c0f62a866 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:29.903733 5332 consensus_queue.cc:260] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:29.904140 5331 consensus_queue.cc:260] T fcff4be357084bf0903cfb1c0f62a866 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:29.904600 5332 ts_tablet_manager.cc:1398] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:29.904947 5332 tablet_bootstrap.cc:492] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:29.904954 5331 ts_tablet_manager.cc:1398] T fcff4be357084bf0903cfb1c0f62a866 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:29.905274 5331 tablet_bootstrap.cc:492] T 7725fa9f98e9439797a532ed6f2f7dba P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:29.907066 5331 tablet_bootstrap.cc:654] T 7725fa9f98e9439797a532ed6f2f7dba P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:29.907078 5332 tablet_bootstrap.cc:654] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:29.910096 5335 tablet_bootstrap.cc:492] T fcff4be357084bf0903cfb1c0f62a866 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:29.910293 5335 ts_tablet_manager.cc:1367] T fcff4be357084bf0903cfb1c0f62a866 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20240501 20:09:29.911290 5335 raft_consensus.cc:348] T fcff4be357084bf0903cfb1c0f62a866 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:29.911605 5335 raft_consensus.cc:374] T fcff4be357084bf0903cfb1c0f62a866 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:29.911707 5335 raft_consensus.cc:729] T fcff4be357084bf0903cfb1c0f62a866 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:29.912062 5335 consensus_queue.cc:260] T fcff4be357084bf0903cfb1c0f62a866 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:29.912771 5335 ts_tablet_manager.cc:1398] T fcff4be357084bf0903cfb1c0f62a866 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:29.913138 5337 tablet_bootstrap.cc:492] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:29.913233 5335 tablet_bootstrap.cc:492] T 0bf6bcf89d444b339612d731af604d0a P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:29.915320 5335 tablet_bootstrap.cc:654] T 0bf6bcf89d444b339612d731af604d0a P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:29.915335 5331 tablet_bootstrap.cc:492] T 7725fa9f98e9439797a532ed6f2f7dba P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:29.915517 5331 ts_tablet_manager.cc:1367] T 7725fa9f98e9439797a532ed6f2f7dba P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240501 20:09:29.916455 5331 raft_consensus.cc:348] T 7725fa9f98e9439797a532ed6f2f7dba P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:29.917594 5331 raft_consensus.cc:374] T 7725fa9f98e9439797a532ed6f2f7dba P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:29.917730 5331 raft_consensus.cc:729] T 7725fa9f98e9439797a532ed6f2f7dba P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:29.918107 5331 consensus_queue.cc:260] T 7725fa9f98e9439797a532ed6f2f7dba P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:29.919211 5331 ts_tablet_manager.cc:1398] T 7725fa9f98e9439797a532ed6f2f7dba P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240501 20:09:29.919687 5331 tablet_bootstrap.cc:492] T 0bf6bcf89d444b339612d731af604d0a P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:29.922371 5337 tablet_bootstrap.cc:654] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:29.922400 5331 tablet_bootstrap.cc:654] T 0bf6bcf89d444b339612d731af604d0a P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:29.924036 5335 tablet_bootstrap.cc:492] T 0bf6bcf89d444b339612d731af604d0a P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:29.924299 5335 ts_tablet_manager.cc:1367] T 0bf6bcf89d444b339612d731af604d0a P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20240501 20:09:29.925626 5335 raft_consensus.cc:348] T 0bf6bcf89d444b339612d731af604d0a P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:29.925951 5335 raft_consensus.cc:374] T 0bf6bcf89d444b339612d731af604d0a P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:29.926064 5335 raft_consensus.cc:729] T 0bf6bcf89d444b339612d731af604d0a P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:29.926488 5335 consensus_queue.cc:260] T 0bf6bcf89d444b339612d731af604d0a P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:29.927574 5335 ts_tablet_manager.cc:1398] T 0bf6bcf89d444b339612d731af604d0a P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 20:09:29.927984 5335 tablet_bootstrap.cc:492] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:29.930485 5335 tablet_bootstrap.cc:654] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:29.930924 5332 tablet_bootstrap.cc:492] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:29.931118 5332 ts_tablet_manager.cc:1367] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.004s
I20240501 20:09:29.932077 5332 raft_consensus.cc:348] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:29.932399 5332 raft_consensus.cc:374] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:29.932482 5332 raft_consensus.cc:729] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:29.932832 5332 consensus_queue.cc:260] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:29.933457 5332 ts_tablet_manager.cc:1398] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:29.934330 5335 tablet_bootstrap.cc:492] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:29.934477 5335 ts_tablet_manager.cc:1367] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 20:09:29.934777 5331 tablet_bootstrap.cc:492] T 0bf6bcf89d444b339612d731af604d0a P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:29.934926 5331 ts_tablet_manager.cc:1367] T 0bf6bcf89d444b339612d731af604d0a P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.004s
I20240501 20:09:29.935937 5331 raft_consensus.cc:348] T 0bf6bcf89d444b339612d731af604d0a P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:29.935526 5335 raft_consensus.cc:348] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:29.936229 5335 raft_consensus.cc:374] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:29.936241 5331 raft_consensus.cc:374] T 0bf6bcf89d444b339612d731af604d0a P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:29.936342 5335 raft_consensus.cc:729] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:29.936348 5331 raft_consensus.cc:729] T 0bf6bcf89d444b339612d731af604d0a P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:29.936612 5331 consensus_queue.cc:260] T 0bf6bcf89d444b339612d731af604d0a P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:29.936692 5335 consensus_queue.cc:260] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:29.937355 5335 ts_tablet_manager.cc:1398] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:29.937433 5331 ts_tablet_manager.cc:1398] T 0bf6bcf89d444b339612d731af604d0a P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:29.942049 5337 tablet_bootstrap.cc:492] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:29.942293 5337 ts_tablet_manager.cc:1367] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.000s
I20240501 20:09:29.943682 5337 raft_consensus.cc:348] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:29.943925 5337 raft_consensus.cc:374] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:29.944041 5337 raft_consensus.cc:729] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:29.944325 5337 consensus_queue.cc:260] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:29.944911 5337 ts_tablet_manager.cc:1398] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:29.965312 5243 raft_consensus.cc:482] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:29.965610 5243 raft_consensus.cc:504] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:29.967821 5243 leader_election.cc:288] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:09:29.976634 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6bcf89d444b339612d731af604d0a" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:29.977015 4499 raft_consensus.cc:2454] T 0bf6bcf89d444b339612d731af604d0a P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:09:29.977102 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6bcf89d444b339612d731af604d0a" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:29.977378 4235 raft_consensus.cc:2454] T 0bf6bcf89d444b339612d731af604d0a P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:09:29.977699 4301 leader_election.cc:302] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:29.978227 5243 raft_consensus.cc:2781] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:29.978382 5243 raft_consensus.cc:482] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:29.978500 5243 raft_consensus.cc:3037] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:29.980753 5243 raft_consensus.cc:504] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:29.981473 5243 leader_election.cc:288] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:09:29.982059 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6bcf89d444b339612d731af604d0a" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:29.982133 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf6bcf89d444b339612d731af604d0a" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:29.982316 4499 raft_consensus.cc:3037] T 0bf6bcf89d444b339612d731af604d0a P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:29.982367 4235 raft_consensus.cc:3037] T 0bf6bcf89d444b339612d731af604d0a P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:29.984450 4235 raft_consensus.cc:2454] T 0bf6bcf89d444b339612d731af604d0a P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:09:29.984459 4499 raft_consensus.cc:2454] T 0bf6bcf89d444b339612d731af604d0a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:09:29.984973 4301 leader_election.cc:302] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:29.985365 5243 raft_consensus.cc:2781] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:29.986016 5243 raft_consensus.cc:686] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Becoming Leader. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Running, Role: LEADER
I20240501 20:09:29.986878 5243 consensus_queue.cc:237] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:29.989272 5241 raft_consensus.cc:482] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:29.989493 5241 raft_consensus.cc:504] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:29.990532 5241 leader_election.cc:288] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:09:29.991254 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7725fa9f98e9439797a532ed6f2f7dba" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:29.991551 4499 raft_consensus.cc:2454] T 7725fa9f98e9439797a532ed6f2f7dba P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:09:29.992412 4087 catalog_manager.cc:5484] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a reported cstate change: term changed from 0 to 1, leader changed from <none> to 10b47782558c41ee91f5e164a372049a (127.29.219.66). New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:29.994906 4564 leader_election.cc:302] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:29.995293 5241 raft_consensus.cc:2781] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:29.995437 5241 raft_consensus.cc:482] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:29.995572 5241 raft_consensus.cc:3037] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:29.998371 5241 raft_consensus.cc:504] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:29.999186 5241 leader_election.cc:288] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:09:29.999382 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7725fa9f98e9439797a532ed6f2f7dba" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:29.999619 4235 raft_consensus.cc:2454] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:09:29.999912 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7725fa9f98e9439797a532ed6f2f7dba" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:29.999904 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7725fa9f98e9439797a532ed6f2f7dba" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:30.000165 4234 raft_consensus.cc:3037] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:30.000187 4499 raft_consensus.cc:3037] T 7725fa9f98e9439797a532ed6f2f7dba P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:30.002359 4234 raft_consensus.cc:2454] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:09:30.002357 4499 raft_consensus.cc:2454] T 7725fa9f98e9439797a532ed6f2f7dba P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:09:30.002874 4564 leader_election.cc:302] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:30.003240 5241 raft_consensus.cc:2781] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:30.003391 5241 raft_consensus.cc:686] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Becoming Leader. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Running, Role: LEADER
I20240501 20:09:30.003686 5241 consensus_queue.cc:237] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:30.006918 4087 catalog_manager.cc:5484] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68). New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:30.216070 5241 raft_consensus.cc:482] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:30.216398 5241 raft_consensus.cc:504] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:30.217494 5241 leader_election.cc:288] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:09:30.218286 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcff4be357084bf0903cfb1c0f62a866" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:30.218366 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcff4be357084bf0903cfb1c0f62a866" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:30.218547 4234 raft_consensus.cc:2454] T fcff4be357084bf0903cfb1c0f62a866 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:09:30.218659 4499 raft_consensus.cc:2454] T fcff4be357084bf0903cfb1c0f62a866 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:09:30.219139 4564 leader_election.cc:302] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:30.219547 5241 raft_consensus.cc:2781] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:30.219791 5241 raft_consensus.cc:482] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:30.220072 5241 raft_consensus.cc:3037] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:30.222458 5241 raft_consensus.cc:504] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:30.223281 5241 leader_election.cc:288] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Requested vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:09:30.224061 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcff4be357084bf0903cfb1c0f62a866" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:30.224064 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcff4be357084bf0903cfb1c0f62a866" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:30.224313 4234 raft_consensus.cc:3037] T fcff4be357084bf0903cfb1c0f62a866 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:30.224349 4499 raft_consensus.cc:3037] T fcff4be357084bf0903cfb1c0f62a866 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:30.227170 4234 raft_consensus.cc:2454] T fcff4be357084bf0903cfb1c0f62a866 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:09:30.227178 4499 raft_consensus.cc:2454] T fcff4be357084bf0903cfb1c0f62a866 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:09:30.227830 4564 leader_election.cc:302] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:30.228236 5241 raft_consensus.cc:2781] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:30.228448 5241 raft_consensus.cc:686] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Becoming Leader. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Running, Role: LEADER
I20240501 20:09:30.228704 5241 consensus_queue.cc:237] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:30.231968 4087 catalog_manager.cc:5484] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68). New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:30.397432 5243 consensus_queue.cc:1035] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:30.405784 5243 consensus_queue.cc:1035] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:30.493085 5241 consensus_queue.cc:1035] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:30.501449 5241 consensus_queue.cc:1035] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:30.659696 5351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:30.660058 5351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:30.668211 5351 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 20:09:30.779338 5241 consensus_queue.cc:1035] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:30.790297 5241 consensus_queue.cc:1035] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 204.537 ms
time per row: 0.102269 ms
-----------------------------------------------------
Suppressions used:
count bytes template
26 6652 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:09:32.002804 4092 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:32.014729 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 63a07bfc290f4bd5a180562344518054 (DEFAULT_TABLE table=rebalance_test_table_2 [id=22f5d6c17f44439eb5b61fdabb03f88f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:32.015326 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a07bfc290f4bd5a180562344518054. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:32.015594 4611 tablet_service.cc:1467] Processing CreateTablet for tablet 54fd2e3ab852426abb0bdd7c3f1b8bfc (DEFAULT_TABLE table=rebalance_test_table_2 [id=22f5d6c17f44439eb5b61fdabb03f88f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:32.015830 4214 tablet_service.cc:1467] Processing CreateTablet for tablet 63a07bfc290f4bd5a180562344518054 (DEFAULT_TABLE table=rebalance_test_table_2 [id=22f5d6c17f44439eb5b61fdabb03f88f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:32.016041 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54fd2e3ab852426abb0bdd7c3f1b8bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:32.016285 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a07bfc290f4bd5a180562344518054. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:32.016745 4347 tablet_service.cc:1467] Processing CreateTablet for tablet 63a07bfc290f4bd5a180562344518054 (DEFAULT_TABLE table=rebalance_test_table_2 [id=22f5d6c17f44439eb5b61fdabb03f88f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:32.017185 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63a07bfc290f4bd5a180562344518054. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:32.019133 4477 tablet_service.cc:1467] Processing CreateTablet for tablet 54fd2e3ab852426abb0bdd7c3f1b8bfc (DEFAULT_TABLE table=rebalance_test_table_2 [id=22f5d6c17f44439eb5b61fdabb03f88f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:32.019618 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54fd2e3ab852426abb0bdd7c3f1b8bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:32.023370 4348 tablet_service.cc:1467] Processing CreateTablet for tablet 54fd2e3ab852426abb0bdd7c3f1b8bfc (DEFAULT_TABLE table=rebalance_test_table_2 [id=22f5d6c17f44439eb5b61fdabb03f88f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:32.023880 4348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54fd2e3ab852426abb0bdd7c3f1b8bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:32.025053 4609 tablet_service.cc:1467] Processing CreateTablet for tablet 4894528c9205457d8586934e822f4cff (DEFAULT_TABLE table=rebalance_test_table_2 [id=22f5d6c17f44439eb5b61fdabb03f88f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:32.025560 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4894528c9205457d8586934e822f4cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:32.028802 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 4894528c9205457d8586934e822f4cff (DEFAULT_TABLE table=rebalance_test_table_2 [id=22f5d6c17f44439eb5b61fdabb03f88f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:32.029287 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4894528c9205457d8586934e822f4cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:32.031064 4346 tablet_service.cc:1467] Processing CreateTablet for tablet 4894528c9205457d8586934e822f4cff (DEFAULT_TABLE table=rebalance_test_table_2 [id=22f5d6c17f44439eb5b61fdabb03f88f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:32.031514 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4894528c9205457d8586934e822f4cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:32.039052 5420 tablet_bootstrap.cc:492] T 63a07bfc290f4bd5a180562344518054 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:32.042477 5420 tablet_bootstrap.cc:654] T 63a07bfc290f4bd5a180562344518054 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:32.042459 5422 tablet_bootstrap.cc:492] T 63a07bfc290f4bd5a180562344518054 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:32.043581 5421 tablet_bootstrap.cc:492] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:32.045701 5421 tablet_bootstrap.cc:654] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:32.047173 5423 tablet_bootstrap.cc:492] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:32.047654 5422 tablet_bootstrap.cc:654] T 63a07bfc290f4bd5a180562344518054 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:32.049232 5423 tablet_bootstrap.cc:654] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:32.049556 5422 tablet_bootstrap.cc:492] T 63a07bfc290f4bd5a180562344518054 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:32.049706 5422 ts_tablet_manager.cc:1367] T 63a07bfc290f4bd5a180562344518054 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 20:09:32.050431 5422 raft_consensus.cc:348] T 63a07bfc290f4bd5a180562344518054 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:32.050763 5422 raft_consensus.cc:374] T 63a07bfc290f4bd5a180562344518054 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:32.050880 5422 raft_consensus.cc:729] T 63a07bfc290f4bd5a180562344518054 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:32.051208 5422 consensus_queue.cc:260] T 63a07bfc290f4bd5a180562344518054 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:32.051589 5423 tablet_bootstrap.cc:492] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:32.051759 5423 ts_tablet_manager.cc:1367] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:09:32.051889 5422 ts_tablet_manager.cc:1398] T 63a07bfc290f4bd5a180562344518054 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:32.052178 5422 tablet_bootstrap.cc:492] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:32.052722 5421 tablet_bootstrap.cc:492] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:32.052585 5423 raft_consensus.cc:348] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:32.052804 5423 raft_consensus.cc:374] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:32.052878 5421 ts_tablet_manager.cc:1367] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240501 20:09:32.052896 5423 raft_consensus.cc:729] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:32.053164 5423 consensus_queue.cc:260] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:32.053776 5421 raft_consensus.cc:348] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:32.053992 5421 raft_consensus.cc:374] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:32.054085 5421 raft_consensus.cc:729] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:32.054358 5421 consensus_queue.cc:260] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:32.056366 5422 tablet_bootstrap.cc:654] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:32.057569 5421 ts_tablet_manager.cc:1398] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:32.057853 5423 ts_tablet_manager.cc:1398] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240501 20:09:32.058072 5420 tablet_bootstrap.cc:492] T 63a07bfc290f4bd5a180562344518054 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:32.058243 5420 ts_tablet_manager.cc:1367] T 63a07bfc290f4bd5a180562344518054 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.008s
I20240501 20:09:32.058461 5421 tablet_bootstrap.cc:492] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:32.059125 5420 raft_consensus.cc:348] T 63a07bfc290f4bd5a180562344518054 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:32.059360 5420 raft_consensus.cc:374] T 63a07bfc290f4bd5a180562344518054 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:32.059463 5420 raft_consensus.cc:729] T 63a07bfc290f4bd5a180562344518054 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:32.059710 5420 consensus_queue.cc:260] T 63a07bfc290f4bd5a180562344518054 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:32.060361 5421 tablet_bootstrap.cc:654] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:32.060388 5420 ts_tablet_manager.cc:1398] T 63a07bfc290f4bd5a180562344518054 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:32.060717 5420 tablet_bootstrap.cc:492] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:32.062196 5420 tablet_bootstrap.cc:654] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:32.062507 5421 tablet_bootstrap.cc:492] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:32.062693 5421 ts_tablet_manager.cc:1367] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240501 20:09:32.063601 5421 raft_consensus.cc:348] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:32.063794 5422 tablet_bootstrap.cc:492] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:32.063864 5421 raft_consensus.cc:374] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:32.063954 5421 raft_consensus.cc:729] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:32.063987 5422 ts_tablet_manager.cc:1367] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240501 20:09:32.064268 5421 consensus_queue.cc:260] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:32.064965 5421 ts_tablet_manager.cc:1398] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:32.065256 5421 tablet_bootstrap.cc:492] T 4894528c9205457d8586934e822f4cff P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:32.065347 5422 raft_consensus.cc:348] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:32.065587 5422 raft_consensus.cc:374] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:32.065690 5422 raft_consensus.cc:729] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:32.066017 5422 consensus_queue.cc:260] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:32.066568 5422 ts_tablet_manager.cc:1398] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:32.066946 5422 tablet_bootstrap.cc:492] T 4894528c9205457d8586934e822f4cff P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:32.067049 5421 tablet_bootstrap.cc:654] T 4894528c9205457d8586934e822f4cff P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:32.068940 5422 tablet_bootstrap.cc:654] T 4894528c9205457d8586934e822f4cff P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:32.075826 5421 tablet_bootstrap.cc:492] T 4894528c9205457d8586934e822f4cff P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:32.075874 5420 tablet_bootstrap.cc:492] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:32.076289 5421 ts_tablet_manager.cc:1367] T 4894528c9205457d8586934e822f4cff P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240501 20:09:32.076280 5420 ts_tablet_manager.cc:1367] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20240501 20:09:32.077350 5420 raft_consensus.cc:348] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:32.077587 5420 raft_consensus.cc:374] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:32.077687 5420 raft_consensus.cc:729] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:32.077561 5421 raft_consensus.cc:348] T 4894528c9205457d8586934e822f4cff P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:32.077860 5421 raft_consensus.cc:374] T 4894528c9205457d8586934e822f4cff P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:32.077992 5421 raft_consensus.cc:729] T 4894528c9205457d8586934e822f4cff P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:32.077996 5420 consensus_queue.cc:260] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:32.078364 5421 consensus_queue.cc:260] T 4894528c9205457d8586934e822f4cff P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:32.079500 5421 ts_tablet_manager.cc:1398] T 4894528c9205457d8586934e822f4cff P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240501 20:09:32.080263 5420 ts_tablet_manager.cc:1398] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20240501 20:09:32.081158 5422 tablet_bootstrap.cc:492] T 4894528c9205457d8586934e822f4cff P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:32.081418 5422 ts_tablet_manager.cc:1367] T 4894528c9205457d8586934e822f4cff P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20240501 20:09:32.082336 5422 raft_consensus.cc:348] T 4894528c9205457d8586934e822f4cff P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:32.082572 5422 raft_consensus.cc:374] T 4894528c9205457d8586934e822f4cff P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:32.082713 5422 raft_consensus.cc:729] T 4894528c9205457d8586934e822f4cff P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:32.082969 5422 consensus_queue.cc:260] T 4894528c9205457d8586934e822f4cff P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:32.083590 5422 ts_tablet_manager.cc:1398] T 4894528c9205457d8586934e822f4cff P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:32.089319 5415 raft_consensus.cc:482] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:32.089519 5415 raft_consensus.cc:504] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:32.090220 5415 leader_election.cc:288] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:32.090729 4367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4894528c9205457d8586934e822f4cff" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:32.090955 4367 raft_consensus.cc:2454] T 4894528c9205457d8586934e822f4cff P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:32.090871 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4894528c9205457d8586934e822f4cff" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:32.091128 4630 raft_consensus.cc:2454] T 4894528c9205457d8586934e822f4cff P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:32.091409 4169 leader_election.cc:302] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:32.091729 5415 raft_consensus.cc:2781] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:32.091864 5415 raft_consensus.cc:482] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:32.091966 5415 raft_consensus.cc:3037] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:32.094128 5415 raft_consensus.cc:504] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:32.094882 5415 leader_election.cc:288] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:32.095620 4367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4894528c9205457d8586934e822f4cff" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:32.095842 4367 raft_consensus.cc:3037] T 4894528c9205457d8586934e822f4cff P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:32.095793 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4894528c9205457d8586934e822f4cff" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:32.096033 4630 raft_consensus.cc:3037] T 4894528c9205457d8586934e822f4cff P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:32.098109 4367 raft_consensus.cc:2454] T 4894528c9205457d8586934e822f4cff P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:32.098306 4630 raft_consensus.cc:2454] T 4894528c9205457d8586934e822f4cff P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:32.098846 4170 leader_election.cc:302] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:32.099288 5415 raft_consensus.cc:2781] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:32.099730 5415 raft_consensus.cc:686] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:09:32.100064 5415 consensus_queue.cc:237] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:32.103125 4085 catalog_manager.cc:5484] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:32.113396 5261 raft_consensus.cc:482] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:32.113629 5261 raft_consensus.cc:504] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:32.114573 5261 leader_election.cc:288] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:32.115139 4367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54fd2e3ab852426abb0bdd7c3f1b8bfc" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:32.115476 4367 raft_consensus.cc:2454] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:32.115967 4434 leader_election.cc:302] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:32.116278 5261 raft_consensus.cc:2781] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:32.116401 5261 raft_consensus.cc:482] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:32.116526 5261 raft_consensus.cc:3037] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:32.118769 5261 raft_consensus.cc:504] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:32.119474 5261 leader_election.cc:288] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:32.119962 4367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54fd2e3ab852426abb0bdd7c3f1b8bfc" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:32.120219 4367 raft_consensus.cc:3037] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:32.121536 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54fd2e3ab852426abb0bdd7c3f1b8bfc" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:32.121708 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54fd2e3ab852426abb0bdd7c3f1b8bfc" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:32.121827 4630 raft_consensus.cc:2454] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:32.122735 4367 raft_consensus.cc:2454] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:32.123143 4434 leader_election.cc:302] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters: 1d34e18a1d394d3f91c9f50d1c7bea4e
I20240501 20:09:32.123518 5261 raft_consensus.cc:2781] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:32.123915 5261 raft_consensus.cc:686] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:09:32.124254 5261 consensus_queue.cc:237] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:32.127297 4085 catalog_manager.cc:5484] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:32.254770 5241 raft_consensus.cc:482] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:32.255003 5241 raft_consensus.cc:504] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:32.255749 5241 leader_election.cc:288] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:32.256307 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a07bfc290f4bd5a180562344518054" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:32.256388 4367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a07bfc290f4bd5a180562344518054" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:32.256527 4235 raft_consensus.cc:2454] T 63a07bfc290f4bd5a180562344518054 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:09:32.256659 4367 raft_consensus.cc:2454] T 63a07bfc290f4bd5a180562344518054 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:09:32.257015 4564 leader_election.cc:302] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:32.257316 5241 raft_consensus.cc:2781] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:32.257424 5241 raft_consensus.cc:482] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:32.257519 5241 raft_consensus.cc:3037] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:32.259728 5241 raft_consensus.cc:504] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:32.260352 5241 leader_election.cc:288] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Requested vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:32.261004 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a07bfc290f4bd5a180562344518054" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:32.261243 4235 raft_consensus.cc:3037] T 63a07bfc290f4bd5a180562344518054 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:32.261466 4367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63a07bfc290f4bd5a180562344518054" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:32.261696 4367 raft_consensus.cc:3037] T 63a07bfc290f4bd5a180562344518054 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:32.264469 4367 raft_consensus.cc:2454] T 63a07bfc290f4bd5a180562344518054 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:09:32.264605 4235 raft_consensus.cc:2454] T 63a07bfc290f4bd5a180562344518054 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:09:32.265060 4564 leader_election.cc:302] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 1d34e18a1d394d3f91c9f50d1c7bea4e; no voters:
I20240501 20:09:32.265555 5241 raft_consensus.cc:2781] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:32.266194 5241 raft_consensus.cc:686] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Becoming Leader. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Running, Role: LEADER
I20240501 20:09:32.266520 5241 consensus_queue.cc:237] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:32.270349 4085 catalog_manager.cc:5484] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68). New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:32.482326 5415 consensus_queue.cc:1035] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:32.488564 5415 consensus_queue.cc:1035] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:32.583731 5261 consensus_queue.cc:1035] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:32.590122 4630 raft_consensus.cc:3037] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:32.593461 5261 consensus_queue.cc:1035] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:32.687057 5441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:32.687397 5441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:32.695353 5441 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 20:09:32.814989 5241 consensus_queue.cc:1035] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:32.874064 4368 raft_consensus.cc:1261] T 63a07bfc290f4bd5a180562344518054 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:32.879192 5489 consensus_queue.cc:1035] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240501 20:09:32.939427 5486 mvcc.cc:205] Tried to move back new op lower bound from 7022977731956289536 to 7022977729606361088. Current Snapshot: MvccSnapshot[applied={T|T < 7022977731956289536}]
INSERT report
rows total: 2000
time total: 234.287 ms
time per row: 0.117144 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:09:34.100322 4089 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:34.112625 4347 tablet_service.cc:1467] Processing CreateTablet for tablet d690199665174c7d9bdbd54d2af4ac70 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5fb9ba92237649f89c99e4487cd45f29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:34.113114 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d690199665174c7d9bdbd54d2af4ac70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:34.113548 4214 tablet_service.cc:1467] Processing CreateTablet for tablet d690199665174c7d9bdbd54d2af4ac70 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5fb9ba92237649f89c99e4487cd45f29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:34.113952 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d690199665174c7d9bdbd54d2af4ac70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:34.117177 4611 tablet_service.cc:1467] Processing CreateTablet for tablet f18af954cefa48d4882313fe18abbd27 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5fb9ba92237649f89c99e4487cd45f29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:34.117777 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f18af954cefa48d4882313fe18abbd27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:34.118351 4215 tablet_service.cc:1467] Processing CreateTablet for tablet f18af954cefa48d4882313fe18abbd27 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5fb9ba92237649f89c99e4487cd45f29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:34.118880 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f18af954cefa48d4882313fe18abbd27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:34.118960 4478 tablet_service.cc:1467] Processing CreateTablet for tablet 23eaac7a50354947b6efa26ac5f01ae7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5fb9ba92237649f89c99e4487cd45f29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:34.119450 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23eaac7a50354947b6efa26ac5f01ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:34.123241 4477 tablet_service.cc:1467] Processing CreateTablet for tablet d690199665174c7d9bdbd54d2af4ac70 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5fb9ba92237649f89c99e4487cd45f29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:34.123699 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d690199665174c7d9bdbd54d2af4ac70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:34.126350 4479 tablet_service.cc:1467] Processing CreateTablet for tablet f18af954cefa48d4882313fe18abbd27 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5fb9ba92237649f89c99e4487cd45f29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:34.126763 4479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f18af954cefa48d4882313fe18abbd27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:34.117177 4609 tablet_service.cc:1467] Processing CreateTablet for tablet 23eaac7a50354947b6efa26ac5f01ae7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5fb9ba92237649f89c99e4487cd45f29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:34.127516 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23eaac7a50354947b6efa26ac5f01ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:34.129279 5515 tablet_bootstrap.cc:492] T d690199665174c7d9bdbd54d2af4ac70 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:34.131141 5513 tablet_bootstrap.cc:492] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:34.132018 5515 tablet_bootstrap.cc:654] T d690199665174c7d9bdbd54d2af4ac70 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:34.133052 5513 tablet_bootstrap.cc:654] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:34.133616 4347 tablet_service.cc:1467] Processing CreateTablet for tablet 23eaac7a50354947b6efa26ac5f01ae7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5fb9ba92237649f89c99e4487cd45f29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:34.134045 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23eaac7a50354947b6efa26ac5f01ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:34.141076 5518 tablet_bootstrap.cc:492] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:34.143857 5518 tablet_bootstrap.cc:654] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:34.144584 5514 tablet_bootstrap.cc:492] T 23eaac7a50354947b6efa26ac5f01ae7 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:34.146266 5514 tablet_bootstrap.cc:654] T 23eaac7a50354947b6efa26ac5f01ae7 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:34.147934 5515 tablet_bootstrap.cc:492] T d690199665174c7d9bdbd54d2af4ac70 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:34.148384 5515 ts_tablet_manager.cc:1367] T d690199665174c7d9bdbd54d2af4ac70 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.015s
I20240501 20:09:34.148577 5513 tablet_bootstrap.cc:492] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:34.148694 5514 tablet_bootstrap.cc:492] T 23eaac7a50354947b6efa26ac5f01ae7 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:34.148753 5513 ts_tablet_manager.cc:1367] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.003s
I20240501 20:09:34.148847 5514 ts_tablet_manager.cc:1367] T 23eaac7a50354947b6efa26ac5f01ae7 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 20:09:34.149569 5513 raft_consensus.cc:348] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:34.149569 5514 raft_consensus.cc:348] T 23eaac7a50354947b6efa26ac5f01ae7 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:34.149793 5514 raft_consensus.cc:374] T 23eaac7a50354947b6efa26ac5f01ae7 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:34.149793 5513 raft_consensus.cc:374] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:34.149891 5514 raft_consensus.cc:729] T 23eaac7a50354947b6efa26ac5f01ae7 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:34.149891 5513 raft_consensus.cc:729] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:34.150120 5514 consensus_queue.cc:260] T 23eaac7a50354947b6efa26ac5f01ae7 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:34.150130 5513 consensus_queue.cc:260] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:34.149654 5515 raft_consensus.cc:348] T d690199665174c7d9bdbd54d2af4ac70 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:34.150707 5513 ts_tablet_manager.cc:1398] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:34.150758 5514 ts_tablet_manager.cc:1398] T 23eaac7a50354947b6efa26ac5f01ae7 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:34.151021 5514 tablet_bootstrap.cc:492] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:34.151031 5513 tablet_bootstrap.cc:492] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:34.151674 5515 raft_consensus.cc:374] T d690199665174c7d9bdbd54d2af4ac70 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:34.151859 5515 raft_consensus.cc:729] T d690199665174c7d9bdbd54d2af4ac70 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:34.152194 5515 consensus_queue.cc:260] T d690199665174c7d9bdbd54d2af4ac70 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:34.152619 5513 tablet_bootstrap.cc:654] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:34.152619 5514 tablet_bootstrap.cc:654] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:34.153100 5515 ts_tablet_manager.cc:1398] T d690199665174c7d9bdbd54d2af4ac70 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 20:09:34.153440 5515 tablet_bootstrap.cc:492] T 23eaac7a50354947b6efa26ac5f01ae7 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:34.155375 5515 tablet_bootstrap.cc:654] T 23eaac7a50354947b6efa26ac5f01ae7 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:34.163971 5515 tablet_bootstrap.cc:492] T 23eaac7a50354947b6efa26ac5f01ae7 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:34.164307 5515 ts_tablet_manager.cc:1367] T 23eaac7a50354947b6efa26ac5f01ae7 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20240501 20:09:34.165345 5515 raft_consensus.cc:348] T 23eaac7a50354947b6efa26ac5f01ae7 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:34.165693 5515 raft_consensus.cc:374] T 23eaac7a50354947b6efa26ac5f01ae7 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:34.165838 5515 raft_consensus.cc:729] T 23eaac7a50354947b6efa26ac5f01ae7 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:34.166203 5515 consensus_queue.cc:260] T 23eaac7a50354947b6efa26ac5f01ae7 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:34.167011 5515 ts_tablet_manager.cc:1398] T 23eaac7a50354947b6efa26ac5f01ae7 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:34.174925 5518 tablet_bootstrap.cc:492] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:34.175237 5518 ts_tablet_manager.cc:1367] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.034s user 0.001s sys 0.010s
I20240501 20:09:34.175904 5513 tablet_bootstrap.cc:492] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:34.176065 5514 tablet_bootstrap.cc:492] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:34.176090 5513 ts_tablet_manager.cc:1367] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.015s
I20240501 20:09:34.176266 5514 ts_tablet_manager.cc:1367] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.019s
I20240501 20:09:34.176560 5518 raft_consensus.cc:348] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:34.176875 5518 raft_consensus.cc:374] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:34.176820 5513 raft_consensus.cc:348] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:34.177047 5513 raft_consensus.cc:374] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:34.177115 5513 raft_consensus.cc:729] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:34.177140 5518 raft_consensus.cc:729] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:34.177337 5513 consensus_queue.cc:260] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:34.177309 5514 raft_consensus.cc:348] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:34.177845 5513 ts_tablet_manager.cc:1398] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:34.177739 5514 raft_consensus.cc:374] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:34.178349 5514 raft_consensus.cc:729] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:34.177642 5518 consensus_queue.cc:260] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:34.179261 5518 ts_tablet_manager.cc:1398] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:34.179636 5518 tablet_bootstrap.cc:492] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:34.181655 5518 tablet_bootstrap.cc:654] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:34.181370 5514 consensus_queue.cc:260] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:34.183570 5514 ts_tablet_manager.cc:1398] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20240501 20:09:34.183931 5514 tablet_bootstrap.cc:492] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:34.185035 5518 tablet_bootstrap.cc:492] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:34.185210 5518 ts_tablet_manager.cc:1367] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 20:09:34.186075 5518 raft_consensus.cc:348] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:34.186224 5514 tablet_bootstrap.cc:654] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:34.186313 5518 raft_consensus.cc:374] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:34.186395 5518 raft_consensus.cc:729] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:34.186698 5518 consensus_queue.cc:260] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:34.187419 5518 ts_tablet_manager.cc:1398] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:34.188769 5491 raft_consensus.cc:482] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:34.188843 5514 tablet_bootstrap.cc:492] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:34.188977 5514 ts_tablet_manager.cc:1367] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240501 20:09:34.188956 5491 raft_consensus.cc:504] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:34.189764 5514 raft_consensus.cc:348] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:34.189997 5514 raft_consensus.cc:374] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:34.189973 5491 leader_election.cc:288] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:34.190091 5514 raft_consensus.cc:729] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:34.190321 5514 consensus_queue.cc:260] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:34.190909 5514 ts_tablet_manager.cc:1398] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:34.191401 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f18af954cefa48d4882313fe18abbd27" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:34.191668 4630 raft_consensus.cc:2454] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:34.192687 4170 leader_election.cc:302] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:34.193158 5491 raft_consensus.cc:2781] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:34.193279 5491 raft_consensus.cc:482] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:34.193373 5491 raft_consensus.cc:3037] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:34.197013 5491 raft_consensus.cc:504] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:34.199661 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f18af954cefa48d4882313fe18abbd27" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:34.199932 4630 raft_consensus.cc:3037] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:34.200670 5491 leader_election.cc:288] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:34.202883 4630 raft_consensus.cc:2454] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:34.203162 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f18af954cefa48d4882313fe18abbd27" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:34.203222 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f18af954cefa48d4882313fe18abbd27" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:34.203352 4170 leader_election.cc:302] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:34.203405 4498 raft_consensus.cc:2454] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:34.203668 5491 raft_consensus.cc:2781] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:34.203825 5491 raft_consensus.cc:686] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:09:34.204087 5491 consensus_queue.cc:237] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:34.206758 4089 catalog_manager.cc:5484] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:34.214176 5491 raft_consensus.cc:482] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:34.214394 5491 raft_consensus.cc:504] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:34.215107 5491 leader_election.cc:288] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:09:34.215670 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d690199665174c7d9bdbd54d2af4ac70" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:34.215740 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d690199665174c7d9bdbd54d2af4ac70" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:34.215909 4368 raft_consensus.cc:2454] T d690199665174c7d9bdbd54d2af4ac70 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:34.216002 4499 raft_consensus.cc:2454] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:34.216356 4169 leader_election.cc:302] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:34.216666 5491 raft_consensus.cc:2781] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:34.216776 5491 raft_consensus.cc:482] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:34.216857 5491 raft_consensus.cc:3037] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:34.218830 5491 raft_consensus.cc:504] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:34.219364 5491 leader_election.cc:288] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:09:34.219900 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d690199665174c7d9bdbd54d2af4ac70" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:34.219955 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d690199665174c7d9bdbd54d2af4ac70" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:34.220106 4368 raft_consensus.cc:3037] T d690199665174c7d9bdbd54d2af4ac70 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:34.220155 4499 raft_consensus.cc:3037] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:34.222018 4368 raft_consensus.cc:2454] T d690199665174c7d9bdbd54d2af4ac70 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:34.222126 4499 raft_consensus.cc:2454] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:34.222425 4169 leader_election.cc:302] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:34.222761 5491 raft_consensus.cc:2781] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:34.222937 5491 raft_consensus.cc:686] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:09:34.223186 5491 consensus_queue.cc:237] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:34.225948 4089 catalog_manager.cc:5484] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:34.255894 5501 raft_consensus.cc:482] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:34.256105 5501 raft_consensus.cc:504] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:34.256727 5501 leader_election.cc:288] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:34.257328 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23eaac7a50354947b6efa26ac5f01ae7" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:34.257304 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23eaac7a50354947b6efa26ac5f01ae7" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:34.257552 4499 raft_consensus.cc:2454] T 23eaac7a50354947b6efa26ac5f01ae7 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:09:34.257572 4368 raft_consensus.cc:2454] T 23eaac7a50354947b6efa26ac5f01ae7 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:09:34.258009 4564 leader_election.cc:302] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 1d34e18a1d394d3f91c9f50d1c7bea4e; no voters:
I20240501 20:09:34.258251 5501 raft_consensus.cc:2781] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:34.258363 5501 raft_consensus.cc:482] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:34.258452 5501 raft_consensus.cc:3037] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:34.260365 5501 raft_consensus.cc:504] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:34.260918 5501 leader_election.cc:288] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:34.261379 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23eaac7a50354947b6efa26ac5f01ae7" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:34.261463 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23eaac7a50354947b6efa26ac5f01ae7" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:34.261564 4499 raft_consensus.cc:3037] T 23eaac7a50354947b6efa26ac5f01ae7 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:34.261698 4368 raft_consensus.cc:3037] T 23eaac7a50354947b6efa26ac5f01ae7 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:34.263510 4499 raft_consensus.cc:2454] T 23eaac7a50354947b6efa26ac5f01ae7 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:09:34.263909 4564 leader_election.cc:302] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:34.264199 5501 raft_consensus.cc:2781] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:34.264333 5501 raft_consensus.cc:686] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Becoming Leader. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Running, Role: LEADER
I20240501 20:09:34.264612 5501 consensus_queue.cc:237] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:34.264873 4368 raft_consensus.cc:2454] T 23eaac7a50354947b6efa26ac5f01ae7 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:09:34.267835 4089 catalog_manager.cc:5484] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68). New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:34.581302 4499 raft_consensus.cc:3037] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:34.584189 5491 consensus_queue.cc:1035] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:34.590165 5528 consensus_queue.cc:1035] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:34.624356 5532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:34.624751 5532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:34.635311 5532 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:09:34.665663 5507 consensus_queue.cc:1035] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:34.670771 5507 consensus_queue.cc:1035] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240501 20:09:34.733548 5528 consensus_queue.cc:1035] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:34.739357 5491 consensus_queue.cc:1035] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 248.204 ms
time per row: 0.124102 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:09:36.003779 4088 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:36.013088 4347 tablet_service.cc:1467] Processing CreateTablet for tablet b72266b07e89450ca96948dd36351dd2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2287807e30d84209930d192b6b5a3d78]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.013561 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b72266b07e89450ca96948dd36351dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.013494 4214 tablet_service.cc:1467] Processing CreateTablet for tablet b72266b07e89450ca96948dd36351dd2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2287807e30d84209930d192b6b5a3d78]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.013866 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b72266b07e89450ca96948dd36351dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.014353 4478 tablet_service.cc:1467] Processing CreateTablet for tablet b72266b07e89450ca96948dd36351dd2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2287807e30d84209930d192b6b5a3d78]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.014780 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b72266b07e89450ca96948dd36351dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.016402 4609 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba181e4f1094e63adf459bf12c7f464 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2287807e30d84209930d192b6b5a3d78]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.016816 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba181e4f1094e63adf459bf12c7f464. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.019009 4611 tablet_service.cc:1467] Processing CreateTablet for tablet 35c0930b782a47c2a74cca36c261a08d (DEFAULT_TABLE table=rebalance_test_table_4 [id=2287807e30d84209930d192b6b5a3d78]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.019384 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35c0930b782a47c2a74cca36c261a08d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.019510 4213 tablet_service.cc:1467] Processing CreateTablet for tablet 35c0930b782a47c2a74cca36c261a08d (DEFAULT_TABLE table=rebalance_test_table_4 [id=2287807e30d84209930d192b6b5a3d78]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.019934 4213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35c0930b782a47c2a74cca36c261a08d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.019850 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba181e4f1094e63adf459bf12c7f464 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2287807e30d84209930d192b6b5a3d78]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.020213 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba181e4f1094e63adf459bf12c7f464. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.019116 5597 tablet_bootstrap.cc:492] T b72266b07e89450ca96948dd36351dd2 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:36.023487 4477 tablet_service.cc:1467] Processing CreateTablet for tablet 35c0930b782a47c2a74cca36c261a08d (DEFAULT_TABLE table=rebalance_test_table_4 [id=2287807e30d84209930d192b6b5a3d78]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.024006 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35c0930b782a47c2a74cca36c261a08d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.027649 4348 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba181e4f1094e63adf459bf12c7f464 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2287807e30d84209930d192b6b5a3d78]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:36.028139 4348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba181e4f1094e63adf459bf12c7f464. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:36.028764 5597 tablet_bootstrap.cc:654] T b72266b07e89450ca96948dd36351dd2 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.031401 5598 tablet_bootstrap.cc:492] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:36.033613 5598 tablet_bootstrap.cc:654] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.042093 5598 tablet_bootstrap.cc:492] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:36.042346 5598 ts_tablet_manager.cc:1367] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240501 20:09:36.043200 5598 raft_consensus.cc:348] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:36.043541 5598 raft_consensus.cc:374] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.043680 5598 raft_consensus.cc:729] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.044202 5598 consensus_queue.cc:260] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:36.047699 5598 ts_tablet_manager.cc:1398] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:36.049566 5598 tablet_bootstrap.cc:492] T 35c0930b782a47c2a74cca36c261a08d P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:36.052870 5598 tablet_bootstrap.cc:654] T 35c0930b782a47c2a74cca36c261a08d P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.056342 5602 tablet_bootstrap.cc:492] T b72266b07e89450ca96948dd36351dd2 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:36.057404 5600 tablet_bootstrap.cc:492] T 1ba181e4f1094e63adf459bf12c7f464 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:36.059476 5600 tablet_bootstrap.cc:654] T 1ba181e4f1094e63adf459bf12c7f464 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.060564 5602 tablet_bootstrap.cc:654] T b72266b07e89450ca96948dd36351dd2 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.061307 5598 tablet_bootstrap.cc:492] T 35c0930b782a47c2a74cca36c261a08d P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:36.061491 5598 ts_tablet_manager.cc:1367] T 35c0930b782a47c2a74cca36c261a08d P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20240501 20:09:36.061841 5600 tablet_bootstrap.cc:492] T 1ba181e4f1094e63adf459bf12c7f464 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:36.062516 5598 raft_consensus.cc:348] T 35c0930b782a47c2a74cca36c261a08d P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:36.062880 5598 raft_consensus.cc:374] T 35c0930b782a47c2a74cca36c261a08d P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.062964 5598 raft_consensus.cc:729] T 35c0930b782a47c2a74cca36c261a08d P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.063246 5598 consensus_queue.cc:260] T 35c0930b782a47c2a74cca36c261a08d P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:36.063824 5598 ts_tablet_manager.cc:1398] T 35c0930b782a47c2a74cca36c261a08d P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:36.064539 5600 ts_tablet_manager.cc:1367] T 1ba181e4f1094e63adf459bf12c7f464 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 20:09:36.065332 5597 tablet_bootstrap.cc:492] T b72266b07e89450ca96948dd36351dd2 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:36.065464 5602 tablet_bootstrap.cc:492] T b72266b07e89450ca96948dd36351dd2 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:36.065508 5597 ts_tablet_manager.cc:1367] T b72266b07e89450ca96948dd36351dd2 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.047s user 0.016s sys 0.020s
I20240501 20:09:36.065447 5600 raft_consensus.cc:348] T 1ba181e4f1094e63adf459bf12c7f464 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:36.065618 5602 ts_tablet_manager.cc:1367] T b72266b07e89450ca96948dd36351dd2 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240501 20:09:36.065659 5600 raft_consensus.cc:374] T 1ba181e4f1094e63adf459bf12c7f464 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.065770 5600 raft_consensus.cc:729] T 1ba181e4f1094e63adf459bf12c7f464 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.066069 5600 consensus_queue.cc:260] T 1ba181e4f1094e63adf459bf12c7f464 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:36.066392 5597 raft_consensus.cc:348] T b72266b07e89450ca96948dd36351dd2 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:36.066442 5602 raft_consensus.cc:348] T b72266b07e89450ca96948dd36351dd2 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:36.066602 5597 raft_consensus.cc:374] T b72266b07e89450ca96948dd36351dd2 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.066659 5602 raft_consensus.cc:374] T b72266b07e89450ca96948dd36351dd2 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.066740 5597 raft_consensus.cc:729] T b72266b07e89450ca96948dd36351dd2 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.066762 5602 raft_consensus.cc:729] T b72266b07e89450ca96948dd36351dd2 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.067008 5602 consensus_queue.cc:260] T b72266b07e89450ca96948dd36351dd2 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:36.067234 5597 consensus_queue.cc:260] T b72266b07e89450ca96948dd36351dd2 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:36.067677 5602 ts_tablet_manager.cc:1398] T b72266b07e89450ca96948dd36351dd2 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:36.068033 5602 tablet_bootstrap.cc:492] T 1ba181e4f1094e63adf459bf12c7f464 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:36.068903 5597 ts_tablet_manager.cc:1398] T b72266b07e89450ca96948dd36351dd2 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:36.069281 5597 tablet_bootstrap.cc:492] T 35c0930b782a47c2a74cca36c261a08d P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:36.070205 5602 tablet_bootstrap.cc:654] T 1ba181e4f1094e63adf459bf12c7f464 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.070286 5600 ts_tablet_manager.cc:1398] T 1ba181e4f1094e63adf459bf12c7f464 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:09:36.071247 5600 tablet_bootstrap.cc:492] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:36.071265 5597 tablet_bootstrap.cc:654] T 35c0930b782a47c2a74cca36c261a08d P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.073405 5600 tablet_bootstrap.cc:654] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.080329 5597 tablet_bootstrap.cc:492] T 35c0930b782a47c2a74cca36c261a08d P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:36.080499 5597 ts_tablet_manager.cc:1367] T 35c0930b782a47c2a74cca36c261a08d P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20240501 20:09:36.080511 5602 tablet_bootstrap.cc:492] T 1ba181e4f1094e63adf459bf12c7f464 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:36.080694 5602 ts_tablet_manager.cc:1367] T 1ba181e4f1094e63adf459bf12c7f464 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240501 20:09:36.081113 5597 raft_consensus.cc:348] T 35c0930b782a47c2a74cca36c261a08d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:36.081401 5597 raft_consensus.cc:374] T 35c0930b782a47c2a74cca36c261a08d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.081521 5597 raft_consensus.cc:729] T 35c0930b782a47c2a74cca36c261a08d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.081768 5602 raft_consensus.cc:348] T 1ba181e4f1094e63adf459bf12c7f464 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:36.081808 5597 consensus_queue.cc:260] T 35c0930b782a47c2a74cca36c261a08d P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:36.081979 5602 raft_consensus.cc:374] T 1ba181e4f1094e63adf459bf12c7f464 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.082067 5602 raft_consensus.cc:729] T 1ba181e4f1094e63adf459bf12c7f464 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.082302 5597 ts_tablet_manager.cc:1398] T 35c0930b782a47c2a74cca36c261a08d P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:36.082311 5602 consensus_queue.cc:260] T 1ba181e4f1094e63adf459bf12c7f464 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:36.082670 5597 tablet_bootstrap.cc:492] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:36.082887 5602 ts_tablet_manager.cc:1398] T 1ba181e4f1094e63adf459bf12c7f464 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:36.083670 5600 tablet_bootstrap.cc:492] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:36.083904 5600 ts_tablet_manager.cc:1367] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20240501 20:09:36.084285 5597 tablet_bootstrap.cc:654] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:36.084738 5600 raft_consensus.cc:348] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:36.084991 5600 raft_consensus.cc:374] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.085089 5600 raft_consensus.cc:729] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.085340 5600 consensus_queue.cc:260] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:36.086066 5600 ts_tablet_manager.cc:1398] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:36.087275 5597 tablet_bootstrap.cc:492] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:36.087450 5597 ts_tablet_manager.cc:1367] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 20:09:36.088322 5597 raft_consensus.cc:348] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:36.088532 5597 raft_consensus.cc:374] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:36.088627 5597 raft_consensus.cc:729] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:36.088850 5597 consensus_queue.cc:260] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:36.089375 5597 ts_tablet_manager.cc:1398] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:36.102960 5528 raft_consensus.cc:482] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:36.103173 5528 raft_consensus.cc:504] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:36.103935 5528 leader_election.cc:288] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:36.104487 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba181e4f1094e63adf459bf12c7f464" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:36.104525 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba181e4f1094e63adf459bf12c7f464" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:36.104718 4368 raft_consensus.cc:2454] T 1ba181e4f1094e63adf459bf12c7f464 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:36.104756 4630 raft_consensus.cc:2454] T 1ba181e4f1094e63adf459bf12c7f464 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:36.105229 4169 leader_election.cc:302] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:36.105500 5528 raft_consensus.cc:2781] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:36.105615 5528 raft_consensus.cc:482] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:36.105710 5528 raft_consensus.cc:3037] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.107714 5528 raft_consensus.cc:504] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:36.108307 5528 leader_election.cc:288] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:36.108769 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba181e4f1094e63adf459bf12c7f464" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:36.108800 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba181e4f1094e63adf459bf12c7f464" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:36.108966 4368 raft_consensus.cc:3037] T 1ba181e4f1094e63adf459bf12c7f464 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.108975 4630 raft_consensus.cc:3037] T 1ba181e4f1094e63adf459bf12c7f464 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.110818 4630 raft_consensus.cc:2454] T 1ba181e4f1094e63adf459bf12c7f464 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:36.111049 4368 raft_consensus.cc:2454] T 1ba181e4f1094e63adf459bf12c7f464 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:36.111281 4170 leader_election.cc:302] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:36.111598 5528 raft_consensus.cc:2781] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:36.112037 5528 raft_consensus.cc:686] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:09:36.112305 5528 consensus_queue.cc:237] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:36.115295 4088 catalog_manager.cc:5484] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:36.167742 5449 raft_consensus.cc:482] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:36.167922 5449 raft_consensus.cc:504] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:36.168659 5449 leader_election.cc:288] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:09:36.169128 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72266b07e89450ca96948dd36351dd2" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:36.169207 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72266b07e89450ca96948dd36351dd2" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:36.169395 4368 raft_consensus.cc:2454] T b72266b07e89450ca96948dd36351dd2 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:36.169451 4235 raft_consensus.cc:2454] T b72266b07e89450ca96948dd36351dd2 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:36.169790 4434 leader_election.cc:302] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:36.170033 5449 raft_consensus.cc:2781] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:36.170141 5449 raft_consensus.cc:482] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:36.170230 5449 raft_consensus.cc:3037] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.172190 5449 raft_consensus.cc:504] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:36.172745 5449 leader_election.cc:288] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:09:36.173221 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72266b07e89450ca96948dd36351dd2" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:36.173296 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b72266b07e89450ca96948dd36351dd2" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:36.173413 4368 raft_consensus.cc:3037] T b72266b07e89450ca96948dd36351dd2 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.173514 4235 raft_consensus.cc:3037] T b72266b07e89450ca96948dd36351dd2 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.175832 4368 raft_consensus.cc:2454] T b72266b07e89450ca96948dd36351dd2 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:36.176218 4434 leader_election.cc:302] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:36.176363 4235 raft_consensus.cc:2454] T b72266b07e89450ca96948dd36351dd2 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:36.176534 5449 raft_consensus.cc:2781] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:36.177290 5449 raft_consensus.cc:686] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:09:36.177570 5449 consensus_queue.cc:237] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:36.180666 4088 catalog_manager.cc:5484] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:36.274359 5501 raft_consensus.cc:482] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:36.274626 5501 raft_consensus.cc:504] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:36.275251 5501 leader_election.cc:288] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:09:36.275782 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c0930b782a47c2a74cca36c261a08d" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:36.275988 4235 raft_consensus.cc:2454] T 35c0930b782a47c2a74cca36c261a08d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:09:36.275910 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c0930b782a47c2a74cca36c261a08d" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:36.276161 4499 raft_consensus.cc:2454] T 35c0930b782a47c2a74cca36c261a08d P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:09:36.276391 4564 leader_election.cc:302] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:36.276650 5501 raft_consensus.cc:2781] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:36.276749 5501 raft_consensus.cc:482] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:36.276855 5501 raft_consensus.cc:3037] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.278856 5501 raft_consensus.cc:504] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:36.279429 5501 leader_election.cc:288] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Requested vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:09:36.279858 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c0930b782a47c2a74cca36c261a08d" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:36.279958 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c0930b782a47c2a74cca36c261a08d" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:36.280052 4235 raft_consensus.cc:3037] T 35c0930b782a47c2a74cca36c261a08d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.280176 4499 raft_consensus.cc:3037] T 35c0930b782a47c2a74cca36c261a08d P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:36.281971 4235 raft_consensus.cc:2454] T 35c0930b782a47c2a74cca36c261a08d P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:09:36.282020 4499 raft_consensus.cc:2454] T 35c0930b782a47c2a74cca36c261a08d P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:09:36.282341 4564 leader_election.cc:302] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:36.282588 5501 raft_consensus.cc:2781] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:36.282989 5501 raft_consensus.cc:686] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Becoming Leader. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Running, Role: LEADER
I20240501 20:09:36.283286 5501 consensus_queue.cc:237] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:36.285740 4088 catalog_manager.cc:5484] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68). New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:36.527963 5528 consensus_queue.cc:1035] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:36.535066 5528 consensus_queue.cc:1035] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:36.603613 5449 consensus_queue.cc:1035] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:36.609511 5449 consensus_queue.cc:1035] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:36.689455 5616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:36.689793 5616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:36.702145 5616 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 20:09:36.830930 5501 consensus_queue.cc:1035] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:36.929473 5501 consensus_queue.cc:1035] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:37.047457 5624 mvcc.cc:205] Tried to move back new op lower bound from 7022977748692873216 to 7022977746058489856. Current Snapshot: MvccSnapshot[applied={T|T < 7022977748692873216 or (T in {7022977748692873216})}]
INSERT report
rows total: 2000
time total: 292.155 ms
time per row: 0.146078 ms
I20240501 20:09:37.146118 4093 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:37.154603 4347 tablet_service.cc:1467] Processing CreateTablet for tablet 41d6e6dce35e4d0da594f147a7f9bfc4 (DEFAULT_TABLE table=rebalance_test_table_5 [id=1e137c7380c64655a3b011211c536d72]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.155131 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41d6e6dce35e4d0da594f147a7f9bfc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.155498 4611 tablet_service.cc:1467] Processing CreateTablet for tablet ddcb6f5090f143c2b672e5630230878f (DEFAULT_TABLE table=rebalance_test_table_5 [id=1e137c7380c64655a3b011211c536d72]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.155966 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddcb6f5090f143c2b672e5630230878f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.156836 4477 tablet_service.cc:1467] Processing CreateTablet for tablet 41d6e6dce35e4d0da594f147a7f9bfc4 (DEFAULT_TABLE table=rebalance_test_table_5 [id=1e137c7380c64655a3b011211c536d72]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.157266 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41d6e6dce35e4d0da594f147a7f9bfc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.157254 4346 tablet_service.cc:1467] Processing CreateTablet for tablet 9d634c1feb7449f686cf3af2a07181e0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=1e137c7380c64655a3b011211c536d72]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.157632 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d634c1feb7449f686cf3af2a07181e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.159003 4348 tablet_service.cc:1467] Processing CreateTablet for tablet ddcb6f5090f143c2b672e5630230878f (DEFAULT_TABLE table=rebalance_test_table_5 [id=1e137c7380c64655a3b011211c536d72]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.159024 4478 tablet_service.cc:1467] Processing CreateTablet for tablet ddcb6f5090f143c2b672e5630230878f (DEFAULT_TABLE table=rebalance_test_table_5 [id=1e137c7380c64655a3b011211c536d72]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.159349 4348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddcb6f5090f143c2b672e5630230878f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.159416 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddcb6f5090f143c2b672e5630230878f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.163010 4479 tablet_service.cc:1467] Processing CreateTablet for tablet 9d634c1feb7449f686cf3af2a07181e0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=1e137c7380c64655a3b011211c536d72]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.163194 4214 tablet_service.cc:1467] Processing CreateTablet for tablet 9d634c1feb7449f686cf3af2a07181e0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=1e137c7380c64655a3b011211c536d72]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.163707 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d634c1feb7449f686cf3af2a07181e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.164602 4609 tablet_service.cc:1467] Processing CreateTablet for tablet 41d6e6dce35e4d0da594f147a7f9bfc4 (DEFAULT_TABLE table=rebalance_test_table_5 [id=1e137c7380c64655a3b011211c536d72]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:37.165041 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41d6e6dce35e4d0da594f147a7f9bfc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.171368 5685 tablet_bootstrap.cc:492] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:37.172951 4479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d634c1feb7449f686cf3af2a07181e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:37.173151 5685 tablet_bootstrap.cc:654] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.180292 5686 tablet_bootstrap.cc:492] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:37.183915 5686 tablet_bootstrap.cc:654] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.203622 5690 tablet_bootstrap.cc:492] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:37.203729 5689 tablet_bootstrap.cc:492] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:37.207561 5690 tablet_bootstrap.cc:654] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.209177 5685 tablet_bootstrap.cc:492] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:37.209407 5685 ts_tablet_manager.cc:1367] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.038s user 0.022s sys 0.013s
I20240501 20:09:37.209583 5689 tablet_bootstrap.cc:654] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.210253 5685 raft_consensus.cc:348] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:37.210575 5685 raft_consensus.cc:374] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.210692 5685 raft_consensus.cc:729] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.210961 5685 consensus_queue.cc:260] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:37.211584 5685 ts_tablet_manager.cc:1398] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:37.211843 5685 tablet_bootstrap.cc:492] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:37.213424 5686 tablet_bootstrap.cc:492] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:37.213483 5685 tablet_bootstrap.cc:654] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.213634 5686 ts_tablet_manager.cc:1367] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.033s user 0.011s sys 0.015s
I20240501 20:09:37.214462 5686 raft_consensus.cc:348] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:37.214816 5686 raft_consensus.cc:374] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.214941 5686 raft_consensus.cc:729] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.215296 5686 consensus_queue.cc:260] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:37.217115 5686 ts_tablet_manager.cc:1398] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:37.217512 5686 tablet_bootstrap.cc:492] T 9d634c1feb7449f686cf3af2a07181e0 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:37.218518 5689 tablet_bootstrap.cc:492] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:37.218755 5689 ts_tablet_manager.cc:1367] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20240501 20:09:37.219615 5689 raft_consensus.cc:348] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:37.219765 5686 tablet_bootstrap.cc:654] T 9d634c1feb7449f686cf3af2a07181e0 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.219930 5689 raft_consensus.cc:374] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.220072 5689 raft_consensus.cc:729] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.220394 5689 consensus_queue.cc:260] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:37.221187 5689 ts_tablet_manager.cc:1398] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:37.221549 5689 tablet_bootstrap.cc:492] T ddcb6f5090f143c2b672e5630230878f P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:37.223306 5689 tablet_bootstrap.cc:654] T ddcb6f5090f143c2b672e5630230878f P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.232789 5690 tablet_bootstrap.cc:492] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:37.233011 5690 ts_tablet_manager.cc:1367] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.005s
I20240501 20:09:37.234113 5690 raft_consensus.cc:348] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:37.234405 5690 raft_consensus.cc:374] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.234505 5690 raft_consensus.cc:729] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.235421 5690 consensus_queue.cc:260] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:37.236378 5690 ts_tablet_manager.cc:1398] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:37.245378 5686 tablet_bootstrap.cc:492] T 9d634c1feb7449f686cf3af2a07181e0 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:37.245456 5685 tablet_bootstrap.cc:492] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:37.245651 5685 ts_tablet_manager.cc:1367] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.024s
I20240501 20:09:37.246162 5686 ts_tablet_manager.cc:1367] T 9d634c1feb7449f686cf3af2a07181e0 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.006s
I20240501 20:09:37.246250 5689 tablet_bootstrap.cc:492] T ddcb6f5090f143c2b672e5630230878f P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:37.246481 5689 ts_tablet_manager.cc:1367] T ddcb6f5090f143c2b672e5630230878f P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.007s
I20240501 20:09:37.246475 5685 raft_consensus.cc:348] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:37.246749 5685 raft_consensus.cc:374] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.246861 5685 raft_consensus.cc:729] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.247004 5686 raft_consensus.cc:348] T 9d634c1feb7449f686cf3af2a07181e0 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:37.247179 5685 consensus_queue.cc:260] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:37.247323 5686 raft_consensus.cc:374] T 9d634c1feb7449f686cf3af2a07181e0 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.247476 5686 raft_consensus.cc:729] T 9d634c1feb7449f686cf3af2a07181e0 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.247435 5689 raft_consensus.cc:348] T ddcb6f5090f143c2b672e5630230878f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:37.247705 5689 raft_consensus.cc:374] T ddcb6f5090f143c2b672e5630230878f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.247787 5685 ts_tablet_manager.cc:1398] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:37.247828 5689 raft_consensus.cc:729] T ddcb6f5090f143c2b672e5630230878f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.247792 5686 consensus_queue.cc:260] T 9d634c1feb7449f686cf3af2a07181e0 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:37.248229 5685 tablet_bootstrap.cc:492] T 9d634c1feb7449f686cf3af2a07181e0 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:37.248135 5689 consensus_queue.cc:260] T ddcb6f5090f143c2b672e5630230878f P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:37.248596 5686 ts_tablet_manager.cc:1398] T 9d634c1feb7449f686cf3af2a07181e0 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:37.248838 5689 ts_tablet_manager.cc:1398] T ddcb6f5090f143c2b672e5630230878f P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:37.250478 5685 tablet_bootstrap.cc:654] T 9d634c1feb7449f686cf3af2a07181e0 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.251252 5686 tablet_bootstrap.cc:492] T ddcb6f5090f143c2b672e5630230878f P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:37.253881 5686 tablet_bootstrap.cc:654] T ddcb6f5090f143c2b672e5630230878f P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:37.257037 5685 tablet_bootstrap.cc:492] T 9d634c1feb7449f686cf3af2a07181e0 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:37.257189 5685 ts_tablet_manager.cc:1367] T 9d634c1feb7449f686cf3af2a07181e0 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.001s
I20240501 20:09:37.257244 5686 tablet_bootstrap.cc:492] T ddcb6f5090f143c2b672e5630230878f P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:37.257395 5686 ts_tablet_manager.cc:1367] T ddcb6f5090f143c2b672e5630230878f P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 20:09:37.257862 5685 raft_consensus.cc:348] T 9d634c1feb7449f686cf3af2a07181e0 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:37.258116 5685 raft_consensus.cc:374] T 9d634c1feb7449f686cf3af2a07181e0 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.258231 5685 raft_consensus.cc:729] T 9d634c1feb7449f686cf3af2a07181e0 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.258456 5685 consensus_queue.cc:260] T 9d634c1feb7449f686cf3af2a07181e0 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:37.258445 5686 raft_consensus.cc:348] T ddcb6f5090f143c2b672e5630230878f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:37.258675 5686 raft_consensus.cc:374] T ddcb6f5090f143c2b672e5630230878f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:37.258771 5686 raft_consensus.cc:729] T ddcb6f5090f143c2b672e5630230878f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:37.258980 5685 ts_tablet_manager.cc:1398] T 9d634c1feb7449f686cf3af2a07181e0 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:37.259032 5686 consensus_queue.cc:260] T ddcb6f5090f143c2b672e5630230878f P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:37.259598 5686 ts_tablet_manager.cc:1398] T ddcb6f5090f143c2b672e5630230878f P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:37.304193 5528 raft_consensus.cc:482] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:37.304421 5528 raft_consensus.cc:504] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:37.305053 5528 leader_election.cc:288] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:09:37.305568 4367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d634c1feb7449f686cf3af2a07181e0" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:37.305797 4367 raft_consensus.cc:2454] T 9d634c1feb7449f686cf3af2a07181e0 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:37.305696 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d634c1feb7449f686cf3af2a07181e0" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:37.305979 4499 raft_consensus.cc:2454] T 9d634c1feb7449f686cf3af2a07181e0 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:37.306310 4169 leader_election.cc:302] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:37.306658 5528 raft_consensus.cc:2781] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:37.306758 5528 raft_consensus.cc:482] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:37.306869 5528 raft_consensus.cc:3037] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.308844 5528 raft_consensus.cc:504] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:37.309389 5528 leader_election.cc:288] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:09:37.309717 4367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d634c1feb7449f686cf3af2a07181e0" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:37.309916 4367 raft_consensus.cc:3037] T 9d634c1feb7449f686cf3af2a07181e0 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.309839 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d634c1feb7449f686cf3af2a07181e0" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:37.310051 4499 raft_consensus.cc:3037] T 9d634c1feb7449f686cf3af2a07181e0 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.312038 4367 raft_consensus.cc:2454] T 9d634c1feb7449f686cf3af2a07181e0 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:37.312078 4499 raft_consensus.cc:2454] T 9d634c1feb7449f686cf3af2a07181e0 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:37.312423 4169 leader_election.cc:302] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:37.312676 5528 raft_consensus.cc:2781] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:37.312829 5528 raft_consensus.cc:686] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:09:37.313094 5528 consensus_queue.cc:237] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:37.315693 4092 catalog_manager.cc:5484] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:37.344211 5658 raft_consensus.cc:482] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:37.344424 5658 raft_consensus.cc:504] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:37.345044 5658 leader_election.cc:288] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:09:37.345661 4367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d6e6dce35e4d0da594f147a7f9bfc4" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:37.345732 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d6e6dce35e4d0da594f147a7f9bfc4" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:37.345865 4367 raft_consensus.cc:2454] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:09:37.345981 4499 raft_consensus.cc:2454] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:09:37.346267 4564 leader_election.cc:302] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 1d34e18a1d394d3f91c9f50d1c7bea4e; no voters:
I20240501 20:09:37.346521 5658 raft_consensus.cc:2781] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:37.346664 5658 raft_consensus.cc:482] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:37.346760 5658 raft_consensus.cc:3037] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.348593 5658 raft_consensus.cc:504] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:37.349118 5658 leader_election.cc:288] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:09:37.349642 4367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d6e6dce35e4d0da594f147a7f9bfc4" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:37.349642 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41d6e6dce35e4d0da594f147a7f9bfc4" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:37.349861 4367 raft_consensus.cc:3037] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.349864 4499 raft_consensus.cc:3037] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.352815 4499 raft_consensus.cc:2454] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:09:37.352809 4367 raft_consensus.cc:2454] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:09:37.353243 4564 leader_election.cc:302] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 1d34e18a1d394d3f91c9f50d1c7bea4e; no voters:
I20240501 20:09:37.353549 5658 raft_consensus.cc:2781] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:37.353688 5658 raft_consensus.cc:686] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Becoming Leader. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Running, Role: LEADER
I20240501 20:09:37.353945 5658 consensus_queue.cc:237] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:37.357245 4091 catalog_manager.cc:5484] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68). New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:37.515861 5617 raft_consensus.cc:482] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:37.516084 5617 raft_consensus.cc:504] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:37.516937 5617 leader_election.cc:288] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:37.517318 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddcb6f5090f143c2b672e5630230878f" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:37.517616 4499 raft_consensus.cc:2454] T ddcb6f5090f143c2b672e5630230878f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:09:37.518108 4301 leader_election.cc:302] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:37.518411 5617 raft_consensus.cc:2781] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:37.518523 5617 raft_consensus.cc:482] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:37.518597 5617 raft_consensus.cc:3037] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.520570 5617 raft_consensus.cc:504] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:37.521217 5617 leader_election.cc:288] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:37.521584 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddcb6f5090f143c2b672e5630230878f" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:37.521832 4499 raft_consensus.cc:3037] T ddcb6f5090f143c2b672e5630230878f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:37.524073 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddcb6f5090f143c2b672e5630230878f" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:37.524142 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddcb6f5090f143c2b672e5630230878f" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:37.524277 4499 raft_consensus.cc:2454] T ddcb6f5090f143c2b672e5630230878f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:09:37.524330 4631 raft_consensus.cc:2454] T ddcb6f5090f143c2b672e5630230878f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:09:37.524693 4301 leader_election.cc:302] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters: 1d34e18a1d394d3f91c9f50d1c7bea4e
I20240501 20:09:37.524984 5617 raft_consensus.cc:2781] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:37.525357 5617 raft_consensus.cc:686] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Becoming Leader. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Running, Role: LEADER
I20240501 20:09:37.525650 5617 consensus_queue.cc:237] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:37.528553 4091 catalog_manager.cc:5484] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a reported cstate change: term changed from 0 to 1, leader changed from <none> to 10b47782558c41ee91f5e164a372049a (127.29.219.66). New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:37.828843 5655 consensus_queue.cc:1035] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:37.853821 5655 consensus_queue.cc:1035] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:37.904662 5528 consensus_queue.cc:1035] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:37.911379 5528 consensus_queue.cc:1035] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:37.981225 5704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:37.981539 5704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:37.988957 5704 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 20:09:38.060730 5617 consensus_queue.cc:1035] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:38.067836 4631 raft_consensus.cc:3037] T ddcb6f5090f143c2b672e5630230878f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:38.073271 5617 consensus_queue.cc:1035] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:38.229235 4170 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.68:45205, user_credentials={real_user=slave}} blocked reactor thread for 83446.8us
INSERT report
rows total: 2000
time total: 273.862 ms
time per row: 0.136931 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:09:39.374259 4092 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:39.383183 4609 tablet_service.cc:1467] Processing CreateTablet for tablet e4b9e63c8c504df198d5774115a57098 (DEFAULT_TABLE table=rebalance_test_table_6 [id=1cba02e90ede4149af49f3f3763a48a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:39.383394 4214 tablet_service.cc:1467] Processing CreateTablet for tablet e4b9e63c8c504df198d5774115a57098 (DEFAULT_TABLE table=rebalance_test_table_6 [id=1cba02e90ede4149af49f3f3763a48a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:39.383438 4346 tablet_service.cc:1467] Processing CreateTablet for tablet e4b9e63c8c504df198d5774115a57098 (DEFAULT_TABLE table=rebalance_test_table_6 [id=1cba02e90ede4149af49f3f3763a48a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:39.383648 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4b9e63c8c504df198d5774115a57098. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:39.383759 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4b9e63c8c504df198d5774115a57098. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:39.383899 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4b9e63c8c504df198d5774115a57098. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:39.385974 4348 tablet_service.cc:1467] Processing CreateTablet for tablet 866614e2559a467ca9f746fa9cfd0f7d (DEFAULT_TABLE table=rebalance_test_table_6 [id=1cba02e90ede4149af49f3f3763a48a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:39.386370 4348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866614e2559a467ca9f746fa9cfd0f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:39.387017 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 866614e2559a467ca9f746fa9cfd0f7d (DEFAULT_TABLE table=rebalance_test_table_6 [id=1cba02e90ede4149af49f3f3763a48a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:39.387370 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866614e2559a467ca9f746fa9cfd0f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:39.388144 4611 tablet_service.cc:1467] Processing CreateTablet for tablet 814749d88a674a278054b44a36a15770 (DEFAULT_TABLE table=rebalance_test_table_6 [id=1cba02e90ede4149af49f3f3763a48a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:39.388576 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 814749d88a674a278054b44a36a15770. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:39.394568 5783 tablet_bootstrap.cc:492] T e4b9e63c8c504df198d5774115a57098 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:39.394812 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 814749d88a674a278054b44a36a15770 (DEFAULT_TABLE table=rebalance_test_table_6 [id=1cba02e90ede4149af49f3f3763a48a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:39.394851 4479 tablet_service.cc:1467] Processing CreateTablet for tablet 814749d88a674a278054b44a36a15770 (DEFAULT_TABLE table=rebalance_test_table_6 [id=1cba02e90ede4149af49f3f3763a48a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:39.395130 4213 tablet_service.cc:1467] Processing CreateTablet for tablet 866614e2559a467ca9f746fa9cfd0f7d (DEFAULT_TABLE table=rebalance_test_table_6 [id=1cba02e90ede4149af49f3f3763a48a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:39.395248 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 814749d88a674a278054b44a36a15770. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:39.395275 4479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 814749d88a674a278054b44a36a15770. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:39.395462 4213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 866614e2559a467ca9f746fa9cfd0f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:39.403141 5784 tablet_bootstrap.cc:492] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:39.403694 5783 tablet_bootstrap.cc:654] T e4b9e63c8c504df198d5774115a57098 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:39.404907 5785 tablet_bootstrap.cc:492] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:39.406951 5784 tablet_bootstrap.cc:654] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:39.407172 5785 tablet_bootstrap.cc:654] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:39.408118 5786 tablet_bootstrap.cc:492] T 814749d88a674a278054b44a36a15770 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:39.411011 5786 tablet_bootstrap.cc:654] T 814749d88a674a278054b44a36a15770 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:39.413405 5783 tablet_bootstrap.cc:492] T e4b9e63c8c504df198d5774115a57098 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:39.413569 5783 ts_tablet_manager.cc:1367] T e4b9e63c8c504df198d5774115a57098 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.006s
I20240501 20:09:39.414026 5785 tablet_bootstrap.cc:492] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:39.414055 5784 tablet_bootstrap.cc:492] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:39.414237 5784 ts_tablet_manager.cc:1367] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20240501 20:09:39.414292 5785 ts_tablet_manager.cc:1367] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20240501 20:09:39.414388 5783 raft_consensus.cc:348] T e4b9e63c8c504df198d5774115a57098 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:39.414606 5783 raft_consensus.cc:374] T e4b9e63c8c504df198d5774115a57098 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:39.414731 5783 raft_consensus.cc:729] T e4b9e63c8c504df198d5774115a57098 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:39.414947 5783 consensus_queue.cc:260] T e4b9e63c8c504df198d5774115a57098 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:39.415473 5783 ts_tablet_manager.cc:1398] T e4b9e63c8c504df198d5774115a57098 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:39.415366 5784 raft_consensus.cc:348] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:39.415777 5783 tablet_bootstrap.cc:492] T 866614e2559a467ca9f746fa9cfd0f7d P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:39.416637 5784 raft_consensus.cc:374] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:39.416797 5784 raft_consensus.cc:729] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:39.416661 5785 raft_consensus.cc:348] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:39.417047 5785 raft_consensus.cc:374] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:39.417182 5785 raft_consensus.cc:729] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:39.417129 5784 consensus_queue.cc:260] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:39.417526 5785 consensus_queue.cc:260] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:39.417723 5783 tablet_bootstrap.cc:654] T 866614e2559a467ca9f746fa9cfd0f7d P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:39.418167 5785 ts_tablet_manager.cc:1398] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 20:09:39.418488 5785 tablet_bootstrap.cc:492] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:39.420719 5785 tablet_bootstrap.cc:654] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:39.423236 5786 tablet_bootstrap.cc:492] T 814749d88a674a278054b44a36a15770 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:39.423437 5786 ts_tablet_manager.cc:1367] T 814749d88a674a278054b44a36a15770 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20240501 20:09:39.424506 5784 ts_tablet_manager.cc:1398] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240501 20:09:39.424366 5786 raft_consensus.cc:348] T 814749d88a674a278054b44a36a15770 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:39.424635 5786 raft_consensus.cc:374] T 814749d88a674a278054b44a36a15770 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:39.424736 5786 raft_consensus.cc:729] T 814749d88a674a278054b44a36a15770 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:39.424798 5784 tablet_bootstrap.cc:492] T 866614e2559a467ca9f746fa9cfd0f7d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:39.424965 5786 consensus_queue.cc:260] T 814749d88a674a278054b44a36a15770 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:39.426064 5786 ts_tablet_manager.cc:1398] T 814749d88a674a278054b44a36a15770 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:39.426506 5784 tablet_bootstrap.cc:654] T 866614e2559a467ca9f746fa9cfd0f7d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:39.429579 5783 tablet_bootstrap.cc:492] T 866614e2559a467ca9f746fa9cfd0f7d P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:39.429755 5783 ts_tablet_manager.cc:1367] T 866614e2559a467ca9f746fa9cfd0f7d P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20240501 20:09:39.430084 5785 tablet_bootstrap.cc:492] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:39.430397 5785 ts_tablet_manager.cc:1367] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20240501 20:09:39.430514 5783 raft_consensus.cc:348] T 866614e2559a467ca9f746fa9cfd0f7d P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:39.430758 5783 raft_consensus.cc:374] T 866614e2559a467ca9f746fa9cfd0f7d P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:39.430864 5783 raft_consensus.cc:729] T 866614e2559a467ca9f746fa9cfd0f7d P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:39.431118 5783 consensus_queue.cc:260] T 866614e2559a467ca9f746fa9cfd0f7d P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:39.431661 5783 ts_tablet_manager.cc:1398] T 866614e2559a467ca9f746fa9cfd0f7d P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:39.431854 5784 tablet_bootstrap.cc:492] T 866614e2559a467ca9f746fa9cfd0f7d P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:39.432014 5784 ts_tablet_manager.cc:1367] T 866614e2559a467ca9f746fa9cfd0f7d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240501 20:09:39.431363 5785 raft_consensus.cc:348] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:39.432133 5785 raft_consensus.cc:374] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:39.432224 5785 raft_consensus.cc:729] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:39.432473 5785 consensus_queue.cc:260] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:39.432868 5784 raft_consensus.cc:348] T 866614e2559a467ca9f746fa9cfd0f7d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:39.433080 5784 raft_consensus.cc:374] T 866614e2559a467ca9f746fa9cfd0f7d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:39.433151 5785 ts_tablet_manager.cc:1398] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:09:39.433149 5784 raft_consensus.cc:729] T 866614e2559a467ca9f746fa9cfd0f7d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:39.433450 5785 tablet_bootstrap.cc:492] T 814749d88a674a278054b44a36a15770 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:39.433365 5784 consensus_queue.cc:260] T 866614e2559a467ca9f746fa9cfd0f7d P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:39.433843 5784 ts_tablet_manager.cc:1398] T 866614e2559a467ca9f746fa9cfd0f7d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:39.434115 5784 tablet_bootstrap.cc:492] T e4b9e63c8c504df198d5774115a57098 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:39.435098 5785 tablet_bootstrap.cc:654] T 814749d88a674a278054b44a36a15770 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:39.435606 5784 tablet_bootstrap.cc:654] T e4b9e63c8c504df198d5774115a57098 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:39.436852 5785 tablet_bootstrap.cc:492] T 814749d88a674a278054b44a36a15770 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:39.436985 5785 ts_tablet_manager.cc:1367] T 814749d88a674a278054b44a36a15770 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:09:39.437881 5785 raft_consensus.cc:348] T 814749d88a674a278054b44a36a15770 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:39.438091 5785 raft_consensus.cc:374] T 814749d88a674a278054b44a36a15770 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:39.438179 5785 raft_consensus.cc:729] T 814749d88a674a278054b44a36a15770 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:39.438382 5785 consensus_queue.cc:260] T 814749d88a674a278054b44a36a15770 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:39.438951 5785 ts_tablet_manager.cc:1398] T 814749d88a674a278054b44a36a15770 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:39.439373 5784 tablet_bootstrap.cc:492] T e4b9e63c8c504df198d5774115a57098 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:39.439507 5784 ts_tablet_manager.cc:1367] T e4b9e63c8c504df198d5774115a57098 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240501 20:09:39.440338 5784 raft_consensus.cc:348] T e4b9e63c8c504df198d5774115a57098 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:39.440544 5784 raft_consensus.cc:374] T e4b9e63c8c504df198d5774115a57098 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:39.440641 5784 raft_consensus.cc:729] T e4b9e63c8c504df198d5774115a57098 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:39.440876 5784 consensus_queue.cc:260] T e4b9e63c8c504df198d5774115a57098 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:39.441437 5784 ts_tablet_manager.cc:1398] T e4b9e63c8c504df198d5774115a57098 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:39.592664 5712 raft_consensus.cc:482] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:39.592842 5712 raft_consensus.cc:504] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:39.593510 5712 leader_election.cc:288] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:39.593974 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b9e63c8c504df198d5774115a57098" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:39.594040 4367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b9e63c8c504df198d5774115a57098" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:39.594269 4631 raft_consensus.cc:2454] T e4b9e63c8c504df198d5774115a57098 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:39.594267 4367 raft_consensus.cc:2454] T e4b9e63c8c504df198d5774115a57098 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:39.594748 4170 leader_election.cc:302] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:39.595026 5712 raft_consensus.cc:2781] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:39.595150 5712 raft_consensus.cc:482] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:39.595227 5712 raft_consensus.cc:3037] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:39.597090 5712 raft_consensus.cc:504] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:39.597638 5712 leader_election.cc:288] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:39.598065 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b9e63c8c504df198d5774115a57098" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:39.598263 4631 raft_consensus.cc:3037] T e4b9e63c8c504df198d5774115a57098 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:39.598227 4367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b9e63c8c504df198d5774115a57098" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:39.598441 4367 raft_consensus.cc:3037] T e4b9e63c8c504df198d5774115a57098 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:39.600152 4631 raft_consensus.cc:2454] T e4b9e63c8c504df198d5774115a57098 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:39.600287 4367 raft_consensus.cc:2454] T e4b9e63c8c504df198d5774115a57098 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:39.600538 4170 leader_election.cc:302] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:39.600847 5712 raft_consensus.cc:2781] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:39.601217 5712 raft_consensus.cc:686] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:09:39.601483 5712 consensus_queue.cc:237] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:39.604382 4091 catalog_manager.cc:5484] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:39.614480 5712 raft_consensus.cc:482] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:39.614730 5712 raft_consensus.cc:504] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:39.615331 5712 leader_election.cc:288] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:39.615903 4367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866614e2559a467ca9f746fa9cfd0f7d" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:39.615880 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866614e2559a467ca9f746fa9cfd0f7d" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:39.616122 4367 raft_consensus.cc:2454] T 866614e2559a467ca9f746fa9cfd0f7d P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:39.616140 4631 raft_consensus.cc:2454] T 866614e2559a467ca9f746fa9cfd0f7d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:39.616608 4170 leader_election.cc:302] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:39.616940 5712 raft_consensus.cc:2781] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:39.617043 5712 raft_consensus.cc:482] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:39.617148 5712 raft_consensus.cc:3037] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:39.619158 5712 raft_consensus.cc:504] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:39.619783 5712 leader_election.cc:288] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:39.620183 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866614e2559a467ca9f746fa9cfd0f7d" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:39.620273 4367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "866614e2559a467ca9f746fa9cfd0f7d" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:39.620409 4631 raft_consensus.cc:3037] T 866614e2559a467ca9f746fa9cfd0f7d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:39.620465 4367 raft_consensus.cc:3037] T 866614e2559a467ca9f746fa9cfd0f7d P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:39.623329 4367 raft_consensus.cc:2454] T 866614e2559a467ca9f746fa9cfd0f7d P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:39.623329 4631 raft_consensus.cc:2454] T 866614e2559a467ca9f746fa9cfd0f7d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:39.623770 4170 leader_election.cc:302] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:39.624049 5712 raft_consensus.cc:2781] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:39.624225 5712 raft_consensus.cc:686] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:09:39.624467 5712 consensus_queue.cc:237] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:39.628603 4091 catalog_manager.cc:5484] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:39.684729 5778 raft_consensus.cc:482] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:39.684899 5778 raft_consensus.cc:504] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:39.685500 5778 leader_election.cc:288] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:09:39.686072 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "814749d88a674a278054b44a36a15770" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:39.686076 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "814749d88a674a278054b44a36a15770" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:39.686301 4498 raft_consensus.cc:2454] T 814749d88a674a278054b44a36a15770 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:09:39.686322 4235 raft_consensus.cc:2454] T 814749d88a674a278054b44a36a15770 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:09:39.686759 4564 leader_election.cc:302] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:39.687019 5778 raft_consensus.cc:2781] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:39.687139 5778 raft_consensus.cc:482] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:39.687223 5778 raft_consensus.cc:3037] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:39.689126 5778 raft_consensus.cc:504] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:39.689622 5778 leader_election.cc:288] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:09:39.690071 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "814749d88a674a278054b44a36a15770" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:39.690135 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "814749d88a674a278054b44a36a15770" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:39.690275 4498 raft_consensus.cc:3037] T 814749d88a674a278054b44a36a15770 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:39.690371 4235 raft_consensus.cc:3037] T 814749d88a674a278054b44a36a15770 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:39.693109 4235 raft_consensus.cc:2454] T 814749d88a674a278054b44a36a15770 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:09:39.693122 4498 raft_consensus.cc:2454] T 814749d88a674a278054b44a36a15770 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:09:39.693526 4564 leader_election.cc:302] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:39.693806 5778 raft_consensus.cc:2781] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:39.693945 5778 raft_consensus.cc:686] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Becoming Leader. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Running, Role: LEADER
I20240501 20:09:39.694239 5778 consensus_queue.cc:237] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:39.697592 4091 catalog_manager.cc:5484] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68). New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:40.019145 5796 consensus_queue.cc:1035] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:40.026657 5796 consensus_queue.cc:1035] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:40.113828 5800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:40.114246 5800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:40.121874 5800 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 20:09:40.179430 5712 consensus_queue.cc:1035] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:40.188074 5796 consensus_queue.cc:1035] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:40.193753 5778 consensus_queue.cc:1035] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:40.213102 5658 consensus_queue.cc:1035] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 276.167 ms
time per row: 0.138083 ms
I20240501 20:09:40.554338 4088 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:40.563242 4214 tablet_service.cc:1467] Processing CreateTablet for tablet f40b062e103d4e54a913fa0f4111f623 (DEFAULT_TABLE table=rebalance_test_table_7 [id=59dd8401cbd048cea30daf86baac8cfa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:40.563452 4479 tablet_service.cc:1467] Processing CreateTablet for tablet f40b062e103d4e54a913fa0f4111f623 (DEFAULT_TABLE table=rebalance_test_table_7 [id=59dd8401cbd048cea30daf86baac8cfa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:40.563766 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f40b062e103d4e54a913fa0f4111f623. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.563863 4479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f40b062e103d4e54a913fa0f4111f623. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.564582 4611 tablet_service.cc:1467] Processing CreateTablet for tablet a8eefa22b6e74b92b9fef036dea46e3b (DEFAULT_TABLE table=rebalance_test_table_7 [id=59dd8401cbd048cea30daf86baac8cfa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:40.564953 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8eefa22b6e74b92b9fef036dea46e3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.567167 4609 tablet_service.cc:1467] Processing CreateTablet for tablet 2aa1a5f266e840698f2127289628dcba (DEFAULT_TABLE table=rebalance_test_table_7 [id=59dd8401cbd048cea30daf86baac8cfa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:40.567669 4348 tablet_service.cc:1467] Processing CreateTablet for tablet 2aa1a5f266e840698f2127289628dcba (DEFAULT_TABLE table=rebalance_test_table_7 [id=59dd8401cbd048cea30daf86baac8cfa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:40.567705 4346 tablet_service.cc:1467] Processing CreateTablet for tablet f40b062e103d4e54a913fa0f4111f623 (DEFAULT_TABLE table=rebalance_test_table_7 [id=59dd8401cbd048cea30daf86baac8cfa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:40.568135 4348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aa1a5f266e840698f2127289628dcba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.571306 4215 tablet_service.cc:1467] Processing CreateTablet for tablet a8eefa22b6e74b92b9fef036dea46e3b (DEFAULT_TABLE table=rebalance_test_table_7 [id=59dd8401cbd048cea30daf86baac8cfa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:40.571362 4477 tablet_service.cc:1467] Processing CreateTablet for tablet 2aa1a5f266e840698f2127289628dcba (DEFAULT_TABLE table=rebalance_test_table_7 [id=59dd8401cbd048cea30daf86baac8cfa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:40.571346 4478 tablet_service.cc:1467] Processing CreateTablet for tablet a8eefa22b6e74b92b9fef036dea46e3b (DEFAULT_TABLE table=rebalance_test_table_7 [id=59dd8401cbd048cea30daf86baac8cfa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:40.571759 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8eefa22b6e74b92b9fef036dea46e3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.571810 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aa1a5f266e840698f2127289628dcba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.573788 5868 tablet_bootstrap.cc:492] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:40.574997 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f40b062e103d4e54a913fa0f4111f623. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.575826 5868 tablet_bootstrap.cc:654] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:40.579438 5870 tablet_bootstrap.cc:492] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:40.579782 5868 tablet_bootstrap.cc:492] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:40.579924 5868 ts_tablet_manager.cc:1367] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240501 20:09:40.580756 5868 raft_consensus.cc:348] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:40.580972 5868 raft_consensus.cc:374] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:40.581054 5868 raft_consensus.cc:729] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:40.581251 5870 tablet_bootstrap.cc:654] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:40.581269 5868 consensus_queue.cc:260] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:40.582113 5868 ts_tablet_manager.cc:1398] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:40.582412 5868 tablet_bootstrap.cc:492] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:40.582449 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aa1a5f266e840698f2127289628dcba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.584100 5868 tablet_bootstrap.cc:654] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:40.591586 5872 tablet_bootstrap.cc:492] T a8eefa22b6e74b92b9fef036dea46e3b P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:40.592588 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8eefa22b6e74b92b9fef036dea46e3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:40.594803 5872 tablet_bootstrap.cc:654] T a8eefa22b6e74b92b9fef036dea46e3b P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:40.596836 5870 tablet_bootstrap.cc:492] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:40.596906 5874 tablet_bootstrap.cc:492] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:40.597002 5870 ts_tablet_manager.cc:1367] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240501 20:09:40.597915 5870 raft_consensus.cc:348] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:40.598148 5870 raft_consensus.cc:374] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:40.598237 5870 raft_consensus.cc:729] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:40.598802 5874 tablet_bootstrap.cc:654] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:40.598532 5870 consensus_queue.cc:260] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:40.601809 5874 tablet_bootstrap.cc:492] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:40.601969 5874 ts_tablet_manager.cc:1367] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 20:09:40.603067 5870 ts_tablet_manager.cc:1398] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240501 20:09:40.604914 5874 raft_consensus.cc:348] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:40.605140 5874 raft_consensus.cc:374] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:40.605247 5874 raft_consensus.cc:729] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:40.605495 5874 consensus_queue.cc:260] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:40.606124 5870 tablet_bootstrap.cc:492] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:40.606235 5874 ts_tablet_manager.cc:1398] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:40.606696 5874 tablet_bootstrap.cc:492] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:40.608423 5874 tablet_bootstrap.cc:654] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:40.608417 5870 tablet_bootstrap.cc:654] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:40.611951 5868 tablet_bootstrap.cc:492] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:40.612214 5868 ts_tablet_manager.cc:1367] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.030s user 0.020s sys 0.009s
I20240501 20:09:40.612330 5874 tablet_bootstrap.cc:492] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:40.612493 5874 ts_tablet_manager.cc:1367] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 20:09:40.612569 5872 tablet_bootstrap.cc:492] T a8eefa22b6e74b92b9fef036dea46e3b P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:40.612730 5872 ts_tablet_manager.cc:1367] T a8eefa22b6e74b92b9fef036dea46e3b P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.004s
I20240501 20:09:40.613266 5868 raft_consensus.cc:348] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:40.613346 5874 raft_consensus.cc:348] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:40.613530 5868 raft_consensus.cc:374] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:40.613541 5874 raft_consensus.cc:374] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:40.613612 5874 raft_consensus.cc:729] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:40.613657 5868 raft_consensus.cc:729] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:40.613593 5872 raft_consensus.cc:348] T a8eefa22b6e74b92b9fef036dea46e3b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:40.613802 5872 raft_consensus.cc:374] T a8eefa22b6e74b92b9fef036dea46e3b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:40.613926 5872 raft_consensus.cc:729] T a8eefa22b6e74b92b9fef036dea46e3b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:40.613948 5868 consensus_queue.cc:260] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:40.613898 5874 consensus_queue.cc:260] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:40.614166 5872 consensus_queue.cc:260] T a8eefa22b6e74b92b9fef036dea46e3b P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:40.614511 5868 ts_tablet_manager.cc:1398] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:40.615236 5874 ts_tablet_manager.cc:1398] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:09:40.616624 5872 ts_tablet_manager.cc:1398] T a8eefa22b6e74b92b9fef036dea46e3b P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 20:09:40.616961 5872 tablet_bootstrap.cc:492] T 2aa1a5f266e840698f2127289628dcba P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:40.618762 5872 tablet_bootstrap.cc:654] T 2aa1a5f266e840698f2127289628dcba P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:40.619021 5870 tablet_bootstrap.cc:492] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:40.619236 5870 ts_tablet_manager.cc:1367] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240501 20:09:40.620106 5870 raft_consensus.cc:348] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:40.620383 5870 raft_consensus.cc:374] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:40.620483 5870 raft_consensus.cc:729] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:40.620769 5870 consensus_queue.cc:260] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:40.621649 5870 ts_tablet_manager.cc:1398] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:40.621989 5870 tablet_bootstrap.cc:492] T a8eefa22b6e74b92b9fef036dea46e3b P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:40.623018 5872 tablet_bootstrap.cc:492] T 2aa1a5f266e840698f2127289628dcba P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:40.623256 5872 ts_tablet_manager.cc:1367] T 2aa1a5f266e840698f2127289628dcba P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240501 20:09:40.623668 5870 tablet_bootstrap.cc:654] T a8eefa22b6e74b92b9fef036dea46e3b P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:40.624292 5872 raft_consensus.cc:348] T 2aa1a5f266e840698f2127289628dcba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:40.624620 5872 raft_consensus.cc:374] T 2aa1a5f266e840698f2127289628dcba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:40.624719 5872 raft_consensus.cc:729] T 2aa1a5f266e840698f2127289628dcba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:40.624997 5872 consensus_queue.cc:260] T 2aa1a5f266e840698f2127289628dcba P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:40.625470 5872 ts_tablet_manager.cc:1398] T 2aa1a5f266e840698f2127289628dcba P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:09:40.626900 5870 tablet_bootstrap.cc:492] T a8eefa22b6e74b92b9fef036dea46e3b P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:40.627080 5870 ts_tablet_manager.cc:1367] T a8eefa22b6e74b92b9fef036dea46e3b P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240501 20:09:40.627763 5870 raft_consensus.cc:348] T a8eefa22b6e74b92b9fef036dea46e3b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:40.628031 5870 raft_consensus.cc:374] T a8eefa22b6e74b92b9fef036dea46e3b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:40.628150 5870 raft_consensus.cc:729] T a8eefa22b6e74b92b9fef036dea46e3b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:40.628410 5870 consensus_queue.cc:260] T a8eefa22b6e74b92b9fef036dea46e3b P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:40.628973 5870 ts_tablet_manager.cc:1398] T a8eefa22b6e74b92b9fef036dea46e3b P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:40.667405 5845 raft_consensus.cc:482] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:40.667591 5845 raft_consensus.cc:504] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:40.668323 5845 leader_election.cc:288] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:40.668825 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa1a5f266e840698f2127289628dcba" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:40.668898 4367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa1a5f266e840698f2127289628dcba" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:40.669066 4631 raft_consensus.cc:2454] T 2aa1a5f266e840698f2127289628dcba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:40.669137 4367 raft_consensus.cc:2454] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:40.669495 4435 leader_election.cc:302] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:40.669816 5845 raft_consensus.cc:2781] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:40.669927 5845 raft_consensus.cc:482] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:40.670035 5845 raft_consensus.cc:3037] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:40.672014 5845 raft_consensus.cc:504] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:40.672672 5845 leader_election.cc:288] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:40.673116 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa1a5f266e840698f2127289628dcba" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:40.673136 4367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa1a5f266e840698f2127289628dcba" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:40.673349 4367 raft_consensus.cc:3037] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:40.673357 4631 raft_consensus.cc:3037] T 2aa1a5f266e840698f2127289628dcba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:40.675419 4631 raft_consensus.cc:2454] T 2aa1a5f266e840698f2127289628dcba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:40.675418 4367 raft_consensus.cc:2454] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:40.675794 4435 leader_election.cc:302] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:40.676093 5845 raft_consensus.cc:2781] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:40.676227 5845 raft_consensus.cc:686] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:09:40.676503 5845 consensus_queue.cc:237] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:40.678233 5857 raft_consensus.cc:482] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:40.678397 5857 raft_consensus.cc:504] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:40.679169 5857 leader_election.cc:288] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:09:40.679529 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8eefa22b6e74b92b9fef036dea46e3b" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:40.679724 4631 raft_consensus.cc:2454] T a8eefa22b6e74b92b9fef036dea46e3b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:40.679806 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8eefa22b6e74b92b9fef036dea46e3b" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:40.679672 4088 catalog_manager.cc:5484] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:40.680017 4498 raft_consensus.cc:2454] T a8eefa22b6e74b92b9fef036dea46e3b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:40.680131 4170 leader_election.cc:302] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:40.680402 5857 raft_consensus.cc:2781] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:40.680490 5857 raft_consensus.cc:482] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:40.680548 5857 raft_consensus.cc:3037] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:40.682750 5857 raft_consensus.cc:504] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:40.683328 5857 leader_election.cc:288] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:09:40.683794 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8eefa22b6e74b92b9fef036dea46e3b" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:40.683977 4631 raft_consensus.cc:3037] T a8eefa22b6e74b92b9fef036dea46e3b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:40.683952 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8eefa22b6e74b92b9fef036dea46e3b" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:40.684139 4498 raft_consensus.cc:3037] T a8eefa22b6e74b92b9fef036dea46e3b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:40.686012 4631 raft_consensus.cc:2454] T a8eefa22b6e74b92b9fef036dea46e3b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:40.686103 4498 raft_consensus.cc:2454] T a8eefa22b6e74b92b9fef036dea46e3b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:40.686473 4170 leader_election.cc:302] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:40.686830 5857 raft_consensus.cc:2781] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:40.686965 5857 raft_consensus.cc:686] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:09:40.687219 5857 consensus_queue.cc:237] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:40.690205 4088 catalog_manager.cc:5484] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:40.824709 5757 raft_consensus.cc:482] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:40.824937 5757 raft_consensus.cc:504] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:40.825690 5757 leader_election.cc:288] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:09:40.826129 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40b062e103d4e54a913fa0f4111f623" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:40.826228 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40b062e103d4e54a913fa0f4111f623" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:40.826350 4235 raft_consensus.cc:2454] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:09:40.826450 4498 raft_consensus.cc:2454] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:09:40.826823 4301 leader_election.cc:302] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:40.827083 5757 raft_consensus.cc:2781] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:40.827203 5757 raft_consensus.cc:482] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:40.827311 5757 raft_consensus.cc:3037] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:40.829339 5757 raft_consensus.cc:504] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:40.829886 5757 leader_election.cc:288] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Requested vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:09:40.830287 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40b062e103d4e54a913fa0f4111f623" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:40.830379 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f40b062e103d4e54a913fa0f4111f623" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:40.830546 4235 raft_consensus.cc:3037] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:40.830576 4498 raft_consensus.cc:3037] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:40.832834 4235 raft_consensus.cc:2454] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:09:40.832935 4498 raft_consensus.cc:2454] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:09:40.833307 4301 leader_election.cc:302] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:40.833806 5757 raft_consensus.cc:2781] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:40.834208 5757 raft_consensus.cc:686] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Becoming Leader. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Running, Role: LEADER
I20240501 20:09:40.834487 5757 consensus_queue.cc:237] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:40.837361 4088 catalog_manager.cc:5484] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a reported cstate change: term changed from 0 to 1, leader changed from <none> to 10b47782558c41ee91f5e164a372049a (127.29.219.66). New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:41.067677 5857 consensus_queue.cc:1035] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.079442 5857 consensus_queue.cc:1035] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.108623 5845 consensus_queue.cc:1035] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.116015 5845 consensus_queue.cc:1035] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.235847 5757 consensus_queue.cc:1035] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:41.240824 5757 consensus_queue.cc:1035] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:41.266122 5885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:41.266510 5885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:41.274528 5885 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
INSERT report
rows total: 2000
time total: 196.714 ms
time per row: 0.0983571 ms
I20240501 20:09:41.618870 4088 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:41.627975 4478 tablet_service.cc:1467] Processing CreateTablet for tablet 85d8684f2bed453c9f0535ce2ac1c6a2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=52695e0315884c0aa878488af56a1006]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:41.628438 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85d8684f2bed453c9f0535ce2ac1c6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.628783 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 85d8684f2bed453c9f0535ce2ac1c6a2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=52695e0315884c0aa878488af56a1006]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:41.629189 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85d8684f2bed453c9f0535ce2ac1c6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.629329 4611 tablet_service.cc:1467] Processing CreateTablet for tablet ff61519287b64c35a573cfca87fc4dc8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=52695e0315884c0aa878488af56a1006]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:41.629727 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff61519287b64c35a573cfca87fc4dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.631013 4609 tablet_service.cc:1467] Processing CreateTablet for tablet e7d2e3f5d38e423dbc905aeabe3dbef8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=52695e0315884c0aa878488af56a1006]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:41.631441 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7d2e3f5d38e423dbc905aeabe3dbef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.633972 4346 tablet_service.cc:1467] Processing CreateTablet for tablet 85d8684f2bed453c9f0535ce2ac1c6a2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=52695e0315884c0aa878488af56a1006]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:41.634379 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85d8684f2bed453c9f0535ce2ac1c6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.636209 4214 tablet_service.cc:1467] Processing CreateTablet for tablet ff61519287b64c35a573cfca87fc4dc8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=52695e0315884c0aa878488af56a1006]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:41.636612 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff61519287b64c35a573cfca87fc4dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.636235 4213 tablet_service.cc:1467] Processing CreateTablet for tablet e7d2e3f5d38e423dbc905aeabe3dbef8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=52695e0315884c0aa878488af56a1006]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:41.636906 4213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7d2e3f5d38e423dbc905aeabe3dbef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.639278 4348 tablet_service.cc:1467] Processing CreateTablet for tablet ff61519287b64c35a573cfca87fc4dc8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=52695e0315884c0aa878488af56a1006]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:41.639657 4348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff61519287b64c35a573cfca87fc4dc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.643689 4477 tablet_service.cc:1467] Processing CreateTablet for tablet e7d2e3f5d38e423dbc905aeabe3dbef8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=52695e0315884c0aa878488af56a1006]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:41.644233 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7d2e3f5d38e423dbc905aeabe3dbef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:41.656314 5947 tablet_bootstrap.cc:492] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:41.657132 5948 tablet_bootstrap.cc:492] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:41.658238 5947 tablet_bootstrap.cc:654] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.658893 5948 tablet_bootstrap.cc:654] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.667532 5948 tablet_bootstrap.cc:492] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:41.668318 5948 ts_tablet_manager.cc:1367] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20240501 20:09:41.669437 5948 raft_consensus.cc:348] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:41.671098 5948 raft_consensus.cc:374] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.671432 5948 raft_consensus.cc:729] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.672063 5948 consensus_queue.cc:260] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:41.673591 5948 ts_tablet_manager.cc:1398] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:41.674584 5948 tablet_bootstrap.cc:492] T ff61519287b64c35a573cfca87fc4dc8 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:41.676041 5953 tablet_bootstrap.cc:492] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:41.678428 5948 tablet_bootstrap.cc:654] T ff61519287b64c35a573cfca87fc4dc8 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.679957 5953 tablet_bootstrap.cc:654] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.681241 5949 tablet_bootstrap.cc:492] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:41.681362 5948 tablet_bootstrap.cc:492] T ff61519287b64c35a573cfca87fc4dc8 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:41.681564 5948 ts_tablet_manager.cc:1367] T ff61519287b64c35a573cfca87fc4dc8 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 20:09:41.682690 5949 tablet_bootstrap.cc:654] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.682579 5948 raft_consensus.cc:348] T ff61519287b64c35a573cfca87fc4dc8 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:41.682922 5948 raft_consensus.cc:374] T ff61519287b64c35a573cfca87fc4dc8 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.683022 5948 raft_consensus.cc:729] T ff61519287b64c35a573cfca87fc4dc8 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.683326 5948 consensus_queue.cc:260] T ff61519287b64c35a573cfca87fc4dc8 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:41.684202 5948 ts_tablet_manager.cc:1398] T ff61519287b64c35a573cfca87fc4dc8 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:41.684551 5948 tablet_bootstrap.cc:492] T e7d2e3f5d38e423dbc905aeabe3dbef8 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:41.685876 5948 tablet_bootstrap.cc:654] T e7d2e3f5d38e423dbc905aeabe3dbef8 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.686924 5949 tablet_bootstrap.cc:492] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:41.687350 5949 ts_tablet_manager.cc:1367] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20240501 20:09:41.689036 5949 raft_consensus.cc:348] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:41.689241 5949 raft_consensus.cc:374] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.689338 5949 raft_consensus.cc:729] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.689558 5949 consensus_queue.cc:260] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:41.689946 5948 tablet_bootstrap.cc:492] T e7d2e3f5d38e423dbc905aeabe3dbef8 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:41.690083 5948 ts_tablet_manager.cc:1367] T e7d2e3f5d38e423dbc905aeabe3dbef8 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 20:09:41.690073 5949 ts_tablet_manager.cc:1398] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:09:41.690337 5949 tablet_bootstrap.cc:492] T ff61519287b64c35a573cfca87fc4dc8 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:41.690697 5948 raft_consensus.cc:348] T e7d2e3f5d38e423dbc905aeabe3dbef8 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:41.691610 5949 tablet_bootstrap.cc:654] T ff61519287b64c35a573cfca87fc4dc8 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.693035 5953 tablet_bootstrap.cc:492] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:41.693807 5947 tablet_bootstrap.cc:492] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:41.693953 5947 ts_tablet_manager.cc:1367] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.038s user 0.015s sys 0.012s
I20240501 20:09:41.694717 5947 raft_consensus.cc:348] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:41.694911 5947 raft_consensus.cc:374] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.694996 5947 raft_consensus.cc:729] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.695189 5947 consensus_queue.cc:260] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:41.695710 5947 ts_tablet_manager.cc:1398] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:41.696066 5953 ts_tablet_manager.cc:1367] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240501 20:09:41.696447 5947 tablet_bootstrap.cc:492] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:41.696964 5953 raft_consensus.cc:348] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:41.697228 5953 raft_consensus.cc:374] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.697352 5953 raft_consensus.cc:729] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.697623 5953 consensus_queue.cc:260] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:41.698421 5953 ts_tablet_manager.cc:1398] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:41.698784 5953 tablet_bootstrap.cc:492] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:41.700466 5947 tablet_bootstrap.cc:654] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.701211 5953 tablet_bootstrap.cc:654] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:41.705637 5949 tablet_bootstrap.cc:492] T ff61519287b64c35a573cfca87fc4dc8 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:41.705895 5949 ts_tablet_manager.cc:1367] T ff61519287b64c35a573cfca87fc4dc8 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.001s
I20240501 20:09:41.711246 5948 raft_consensus.cc:374] T e7d2e3f5d38e423dbc905aeabe3dbef8 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.711413 5948 raft_consensus.cc:729] T e7d2e3f5d38e423dbc905aeabe3dbef8 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.711650 5947 tablet_bootstrap.cc:492] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:41.711834 5947 ts_tablet_manager.cc:1367] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.008s
I20240501 20:09:41.711759 5948 consensus_queue.cc:260] T e7d2e3f5d38e423dbc905aeabe3dbef8 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:41.712997 5947 raft_consensus.cc:348] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:41.713258 5947 raft_consensus.cc:374] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.713347 5947 raft_consensus.cc:729] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.713591 5947 consensus_queue.cc:260] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:41.714830 5947 ts_tablet_manager.cc:1398] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:41.715237 5948 ts_tablet_manager.cc:1398] T e7d2e3f5d38e423dbc905aeabe3dbef8 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240501 20:09:41.715184 5949 raft_consensus.cc:348] T ff61519287b64c35a573cfca87fc4dc8 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:41.715405 5949 raft_consensus.cc:374] T ff61519287b64c35a573cfca87fc4dc8 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.715482 5949 raft_consensus.cc:729] T ff61519287b64c35a573cfca87fc4dc8 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.715690 5949 consensus_queue.cc:260] T ff61519287b64c35a573cfca87fc4dc8 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:41.716137 5949 ts_tablet_manager.cc:1398] T ff61519287b64c35a573cfca87fc4dc8 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240501 20:09:41.726897 5953 tablet_bootstrap.cc:492] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:41.727218 5953 ts_tablet_manager.cc:1367] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.028s user 0.018s sys 0.000s
I20240501 20:09:41.727978 5953 raft_consensus.cc:348] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:41.728462 5953 raft_consensus.cc:374] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:41.728549 5953 raft_consensus.cc:729] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:41.728952 5953 consensus_queue.cc:260] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:41.729498 5953 ts_tablet_manager.cc:1398] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:41.780543 5796 raft_consensus.cc:482] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.780771 5796 raft_consensus.cc:504] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:41.781399 5796 leader_election.cc:288] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:41.781944 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d8684f2bed453c9f0535ce2ac1c6a2" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:41.781944 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d8684f2bed453c9f0535ce2ac1c6a2" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:41.782172 4368 raft_consensus.cc:2454] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:41.782172 4499 raft_consensus.cc:2454] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:41.782552 4169 leader_election.cc:302] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:41.782851 5796 raft_consensus.cc:2781] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:41.782950 5796 raft_consensus.cc:482] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:41.783052 5796 raft_consensus.cc:3037] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.784858 5796 raft_consensus.cc:504] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:41.785398 5796 leader_election.cc:288] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:41.785840 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d8684f2bed453c9f0535ce2ac1c6a2" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:41.785912 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85d8684f2bed453c9f0535ce2ac1c6a2" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:41.786074 4499 raft_consensus.cc:3037] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.786115 4368 raft_consensus.cc:3037] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.789076 4368 raft_consensus.cc:2454] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:41.789076 4499 raft_consensus.cc:2454] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:41.789582 4169 leader_election.cc:302] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:41.789882 5796 raft_consensus.cc:2781] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:41.790050 5796 raft_consensus.cc:686] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:09:41.790319 5796 consensus_queue.cc:237] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:41.793435 4088 catalog_manager.cc:5484] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:41.950225 5896 raft_consensus.cc:482] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.950443 5896 raft_consensus.cc:504] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:41.951138 5896 leader_election.cc:288] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:41.951699 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff61519287b64c35a573cfca87fc4dc8" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:41.951684 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff61519287b64c35a573cfca87fc4dc8" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:41.951961 4368 raft_consensus.cc:2454] T ff61519287b64c35a573cfca87fc4dc8 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:09:41.951979 4235 raft_consensus.cc:2454] T ff61519287b64c35a573cfca87fc4dc8 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:09:41.952364 4564 leader_election.cc:302] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 1d34e18a1d394d3f91c9f50d1c7bea4e; no voters:
I20240501 20:09:41.952607 5896 raft_consensus.cc:2781] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:41.952725 5896 raft_consensus.cc:482] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:41.952824 5896 raft_consensus.cc:3037] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.954665 5931 raft_consensus.cc:482] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:41.954810 5931 raft_consensus.cc:504] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:41.954882 5896 raft_consensus.cc:504] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:41.955525 5896 leader_election.cc:288] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Requested vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:41.955546 5931 leader_election.cc:288] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:09:41.956029 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7d2e3f5d38e423dbc905aeabe3dbef8" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:41.956125 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff61519287b64c35a573cfca87fc4dc8" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:41.956185 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7d2e3f5d38e423dbc905aeabe3dbef8" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:41.956288 4235 raft_consensus.cc:2454] T e7d2e3f5d38e423dbc905aeabe3dbef8 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:41.956328 4233 raft_consensus.cc:3037] T ff61519287b64c35a573cfca87fc4dc8 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.956403 4631 raft_consensus.cc:2454] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:41.956669 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff61519287b64c35a573cfca87fc4dc8" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:41.956780 4435 leader_election.cc:302] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:41.956905 4368 raft_consensus.cc:3037] T ff61519287b64c35a573cfca87fc4dc8 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.958590 4233 raft_consensus.cc:2454] T ff61519287b64c35a573cfca87fc4dc8 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:09:41.958931 5931 raft_consensus.cc:2781] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:41.959064 5931 raft_consensus.cc:482] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:41.959050 4564 leader_election.cc:302] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:41.959172 5931 raft_consensus.cc:3037] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.959362 5896 raft_consensus.cc:2781] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:41.959460 4368 raft_consensus.cc:2454] T ff61519287b64c35a573cfca87fc4dc8 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:09:41.959854 5896 raft_consensus.cc:686] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Becoming Leader. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Running, Role: LEADER
I20240501 20:09:41.960340 5896 consensus_queue.cc:237] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:41.961917 5931 raft_consensus.cc:504] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:41.962604 5931 leader_election.cc:288] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:09:41.962916 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7d2e3f5d38e423dbc905aeabe3dbef8" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:41.963155 4233 raft_consensus.cc:3037] T e7d2e3f5d38e423dbc905aeabe3dbef8 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.963631 4088 catalog_manager.cc:5484] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68). New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:41.964037 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7d2e3f5d38e423dbc905aeabe3dbef8" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:41.964231 4631 raft_consensus.cc:3037] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:41.965596 4233 raft_consensus.cc:2454] T e7d2e3f5d38e423dbc905aeabe3dbef8 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:41.966148 4434 leader_election.cc:302] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:41.966450 5931 raft_consensus.cc:2781] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:41.966572 5931 raft_consensus.cc:686] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:09:41.966890 5931 consensus_queue.cc:237] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:41.967176 4631 raft_consensus.cc:2454] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:41.969763 4088 catalog_manager.cc:5484] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:42.243062 5796 consensus_queue.cc:1035] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:42.250945 5857 consensus_queue.cc:1035] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:42.357843 5962 consensus_queue.cc:1035] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:42.364099 5962 consensus_queue.cc:1035] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:42.397950 5965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:42.398275 5965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:42.406533 5965 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 20:09:42.480026 5845 consensus_queue.cc:1035] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:42.511656 6001 consensus_queue.cc:1035] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
INSERT report
rows total: 2000
time total: 295.748 ms
time per row: 0.147874 ms
I20240501 20:09:42.852280 4083 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:42.862052 4477 tablet_service.cc:1467] Processing CreateTablet for tablet ef7c0a48f11945d3953f3461d06bc184 (DEFAULT_TABLE table=rebalance_test_table_9 [id=eef6790ef9874038b63d81f758f188d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:42.862526 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef7c0a48f11945d3953f3461d06bc184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.862715 4478 tablet_service.cc:1467] Processing CreateTablet for tablet 65b0048212a745c08ab24ead346bf74e (DEFAULT_TABLE table=rebalance_test_table_9 [id=eef6790ef9874038b63d81f758f188d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:42.863143 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b0048212a745c08ab24ead346bf74e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.863502 4346 tablet_service.cc:1467] Processing CreateTablet for tablet ef7c0a48f11945d3953f3461d06bc184 (DEFAULT_TABLE table=rebalance_test_table_9 [id=eef6790ef9874038b63d81f758f188d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:42.863673 4347 tablet_service.cc:1467] Processing CreateTablet for tablet 644d4d293e8c462297c6419965bd2c89 (DEFAULT_TABLE table=rebalance_test_table_9 [id=eef6790ef9874038b63d81f758f188d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:42.863942 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef7c0a48f11945d3953f3461d06bc184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.864183 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 644d4d293e8c462297c6419965bd2c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.864209 4611 tablet_service.cc:1467] Processing CreateTablet for tablet 65b0048212a745c08ab24ead346bf74e (DEFAULT_TABLE table=rebalance_test_table_9 [id=eef6790ef9874038b63d81f758f188d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:42.864594 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b0048212a745c08ab24ead346bf74e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.864554 4348 tablet_service.cc:1467] Processing CreateTablet for tablet 65b0048212a745c08ab24ead346bf74e (DEFAULT_TABLE table=rebalance_test_table_9 [id=eef6790ef9874038b63d81f758f188d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:42.864892 4348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b0048212a745c08ab24ead346bf74e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.866168 4609 tablet_service.cc:1467] Processing CreateTablet for tablet ef7c0a48f11945d3953f3461d06bc184 (DEFAULT_TABLE table=rebalance_test_table_9 [id=eef6790ef9874038b63d81f758f188d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:42.866564 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef7c0a48f11945d3953f3461d06bc184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.870117 4213 tablet_service.cc:1467] Processing CreateTablet for tablet 644d4d293e8c462297c6419965bd2c89 (DEFAULT_TABLE table=rebalance_test_table_9 [id=eef6790ef9874038b63d81f758f188d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:42.870222 4479 tablet_service.cc:1467] Processing CreateTablet for tablet 644d4d293e8c462297c6419965bd2c89 (DEFAULT_TABLE table=rebalance_test_table_9 [id=eef6790ef9874038b63d81f758f188d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:42.870636 4213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 644d4d293e8c462297c6419965bd2c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.870642 4479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 644d4d293e8c462297c6419965bd2c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:42.879532 6032 tablet_bootstrap.cc:492] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:42.882958 6032 tablet_bootstrap.cc:654] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:42.888027 6030 tablet_bootstrap.cc:492] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:42.892625 6030 tablet_bootstrap.cc:654] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:42.892799 6031 tablet_bootstrap.cc:492] T 644d4d293e8c462297c6419965bd2c89 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:42.894323 6034 tablet_bootstrap.cc:492] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:42.895155 6032 tablet_bootstrap.cc:492] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:42.895359 6032 ts_tablet_manager.cc:1367] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20240501 20:09:42.896488 6034 tablet_bootstrap.cc:654] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:42.897827 6031 tablet_bootstrap.cc:654] T 644d4d293e8c462297c6419965bd2c89 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:42.897886 6032 raft_consensus.cc:348] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:42.898234 6032 raft_consensus.cc:374] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:42.898398 6032 raft_consensus.cc:729] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:42.898809 6032 consensus_queue.cc:260] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:42.899747 6032 ts_tablet_manager.cc:1398] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:09:42.900152 6032 tablet_bootstrap.cc:492] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:42.901492 6030 tablet_bootstrap.cc:492] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:42.901759 6030 ts_tablet_manager.cc:1367] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20240501 20:09:42.902105 6032 tablet_bootstrap.cc:654] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:42.902787 6030 raft_consensus.cc:348] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:42.903075 6030 raft_consensus.cc:374] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:42.903229 6030 raft_consensus.cc:729] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:42.903975 6030 consensus_queue.cc:260] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:42.906675 6030 ts_tablet_manager.cc:1398] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:42.907781 6030 tablet_bootstrap.cc:492] T ef7c0a48f11945d3953f3461d06bc184 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:42.910321 6030 tablet_bootstrap.cc:654] T ef7c0a48f11945d3953f3461d06bc184 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:42.919683 6031 tablet_bootstrap.cc:492] T 644d4d293e8c462297c6419965bd2c89 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:42.920526 6031 ts_tablet_manager.cc:1367] T 644d4d293e8c462297c6419965bd2c89 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.000s
I20240501 20:09:42.921779 6031 raft_consensus.cc:348] T 644d4d293e8c462297c6419965bd2c89 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:42.922200 6031 raft_consensus.cc:374] T 644d4d293e8c462297c6419965bd2c89 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:42.922364 6031 raft_consensus.cc:729] T 644d4d293e8c462297c6419965bd2c89 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:42.922778 6031 consensus_queue.cc:260] T 644d4d293e8c462297c6419965bd2c89 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:42.924006 6031 ts_tablet_manager.cc:1398] T 644d4d293e8c462297c6419965bd2c89 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:42.924496 6031 tablet_bootstrap.cc:492] T ef7c0a48f11945d3953f3461d06bc184 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:42.924942 6032 tablet_bootstrap.cc:492] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:42.925170 6032 ts_tablet_manager.cc:1367] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.010s
I20240501 20:09:42.925985 6032 raft_consensus.cc:348] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:42.926170 6032 raft_consensus.cc:374] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:42.926249 6032 raft_consensus.cc:729] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:42.926319 6034 tablet_bootstrap.cc:492] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:42.926487 6032 consensus_queue.cc:260] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:42.926741 6034 ts_tablet_manager.cc:1367] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.028s
I20240501 20:09:42.927997 6032 ts_tablet_manager.cc:1398] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:09:42.928385 6032 tablet_bootstrap.cc:492] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:42.929172 6030 tablet_bootstrap.cc:492] T ef7c0a48f11945d3953f3461d06bc184 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:42.929463 6030 ts_tablet_manager.cc:1367] T ef7c0a48f11945d3953f3461d06bc184 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.007s
I20240501 20:09:42.930310 6030 raft_consensus.cc:348] T ef7c0a48f11945d3953f3461d06bc184 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:42.930670 6030 raft_consensus.cc:374] T ef7c0a48f11945d3953f3461d06bc184 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:42.930799 6030 raft_consensus.cc:729] T ef7c0a48f11945d3953f3461d06bc184 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:42.931129 6030 consensus_queue.cc:260] T ef7c0a48f11945d3953f3461d06bc184 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:42.931461 6032 tablet_bootstrap.cc:654] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:42.931519 6031 tablet_bootstrap.cc:654] T ef7c0a48f11945d3953f3461d06bc184 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:42.934124 6034 raft_consensus.cc:348] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:42.934564 6034 raft_consensus.cc:374] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:42.934789 6034 raft_consensus.cc:729] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:42.935575 6030 ts_tablet_manager.cc:1398] T ef7c0a48f11945d3953f3461d06bc184 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 20:09:42.937392 6034 consensus_queue.cc:260] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:42.938606 6034 ts_tablet_manager.cc:1398] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 20:09:42.939858 6031 tablet_bootstrap.cc:492] T ef7c0a48f11945d3953f3461d06bc184 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:42.940133 6031 ts_tablet_manager.cc:1367] T ef7c0a48f11945d3953f3461d06bc184 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20240501 20:09:42.941470 6031 raft_consensus.cc:348] T ef7c0a48f11945d3953f3461d06bc184 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:42.941812 6031 raft_consensus.cc:374] T ef7c0a48f11945d3953f3461d06bc184 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:42.941942 6031 raft_consensus.cc:729] T ef7c0a48f11945d3953f3461d06bc184 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:42.943532 6031 consensus_queue.cc:260] T ef7c0a48f11945d3953f3461d06bc184 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:42.944379 6031 ts_tablet_manager.cc:1398] T ef7c0a48f11945d3953f3461d06bc184 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 20:09:42.944669 6031 tablet_bootstrap.cc:492] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:42.946135 6032 tablet_bootstrap.cc:492] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:42.946290 6032 ts_tablet_manager.cc:1367] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.011s
I20240501 20:09:42.946421 6031 tablet_bootstrap.cc:654] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:42.947113 6032 raft_consensus.cc:348] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:42.947413 6032 raft_consensus.cc:374] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:42.947582 6032 raft_consensus.cc:729] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:42.947929 6032 consensus_queue.cc:260] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:42.948494 6032 ts_tablet_manager.cc:1398] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:42.950912 6031 tablet_bootstrap.cc:492] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:42.951072 6031 ts_tablet_manager.cc:1367] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20240501 20:09:42.951673 6031 raft_consensus.cc:348] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:42.951849 6031 raft_consensus.cc:374] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:42.951943 6031 raft_consensus.cc:729] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:42.952174 6031 consensus_queue.cc:260] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:42.952708 6031 ts_tablet_manager.cc:1398] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:42.969099 5999 raft_consensus.cc:482] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:42.969277 5999 raft_consensus.cc:504] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:42.969867 5999 leader_election.cc:288] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:42.970392 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "644d4d293e8c462297c6419965bd2c89" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:42.970465 4367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "644d4d293e8c462297c6419965bd2c89" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:42.970710 4367 raft_consensus.cc:2454] T 644d4d293e8c462297c6419965bd2c89 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:42.970706 4235 raft_consensus.cc:2454] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:42.971176 4434 leader_election.cc:302] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:42.971458 5999 raft_consensus.cc:2781] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:42.971565 5999 raft_consensus.cc:482] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:42.971652 5999 raft_consensus.cc:3037] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:42.973716 5999 raft_consensus.cc:504] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:42.974169 6002 raft_consensus.cc:482] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:42.974282 5999 leader_election.cc:288] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Requested vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:42.974340 6002 raft_consensus.cc:504] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:42.974838 4367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "644d4d293e8c462297c6419965bd2c89" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:42.975054 4367 raft_consensus.cc:3037] T 644d4d293e8c462297c6419965bd2c89 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:42.975147 6002 leader_election.cc:288] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:42.975445 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "644d4d293e8c462297c6419965bd2c89" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:42.975524 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7c0a48f11945d3953f3461d06bc184" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:42.975661 4235 raft_consensus.cc:3037] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:42.975694 4631 raft_consensus.cc:2454] T ef7c0a48f11945d3953f3461d06bc184 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:42.976083 4435 leader_election.cc:302] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:42.976296 6002 raft_consensus.cc:2781] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:42.976370 6002 raft_consensus.cc:482] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:42.976439 6002 raft_consensus.cc:3037] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:42.977982 4367 raft_consensus.cc:2454] T 644d4d293e8c462297c6419965bd2c89 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:42.978380 4235 raft_consensus.cc:2454] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:42.978397 6002 raft_consensus.cc:504] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:42.978487 4434 leader_election.cc:302] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:42.978813 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7c0a48f11945d3953f3461d06bc184" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:42.979029 4368 raft_consensus.cc:2454] T ef7c0a48f11945d3953f3461d06bc184 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:42.979048 6002 leader_election.cc:288] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:42.979228 6002 raft_consensus.cc:2781] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:42.979364 6002 raft_consensus.cc:686] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:09:42.979579 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7c0a48f11945d3953f3461d06bc184" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:42.979784 4631 raft_consensus.cc:3037] T ef7c0a48f11945d3953f3461d06bc184 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:42.979666 6002 consensus_queue.cc:237] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:42.979851 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7c0a48f11945d3953f3461d06bc184" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:42.980012 4368 raft_consensus.cc:3037] T ef7c0a48f11945d3953f3461d06bc184 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:42.982496 4631 raft_consensus.cc:2454] T ef7c0a48f11945d3953f3461d06bc184 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:42.982575 4368 raft_consensus.cc:2454] T ef7c0a48f11945d3953f3461d06bc184 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:42.982985 4435 leader_election.cc:302] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:42.983265 6002 raft_consensus.cc:2781] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:42.983211 4084 catalog_manager.cc:5484] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:42.983491 6002 raft_consensus.cc:686] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:09:42.983714 6002 consensus_queue.cc:237] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:42.988018 4084 catalog_manager.cc:5484] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:43.069334 6002 raft_consensus.cc:482] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:43.069542 6002 raft_consensus.cc:504] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:43.070207 6002 leader_election.cc:288] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:43.070824 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b0048212a745c08ab24ead346bf74e" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:43.070817 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b0048212a745c08ab24ead346bf74e" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:43.071069 4368 raft_consensus.cc:2454] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:43.071094 4631 raft_consensus.cc:2454] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:43.071491 4434 leader_election.cc:302] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:43.071779 6002 raft_consensus.cc:2781] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:43.071875 6002 raft_consensus.cc:482] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:43.071957 6002 raft_consensus.cc:3037] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:43.073799 6002 raft_consensus.cc:504] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:43.074366 6002 leader_election.cc:288] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:43.075153 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b0048212a745c08ab24ead346bf74e" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:43.075160 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b0048212a745c08ab24ead346bf74e" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:43.075388 4631 raft_consensus.cc:3037] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:43.075388 4368 raft_consensus.cc:3037] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:43.078178 4631 raft_consensus.cc:2454] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:43.078179 4368 raft_consensus.cc:2454] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:43.078579 4435 leader_election.cc:302] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:43.078902 6002 raft_consensus.cc:2781] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:43.079053 6002 raft_consensus.cc:686] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:09:43.079311 6002 consensus_queue.cc:237] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:43.082237 4084 catalog_manager.cc:5484] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:43.419298 5999 consensus_queue.cc:1035] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:43.429919 6002 consensus_queue.cc:1035] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:43.458644 6002 consensus_queue.cc:1035] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:43.463835 6002 consensus_queue.cc:1035] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:43.500967 6047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:43.501297 6047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:43.508893 6047 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:09:43.535252 6054 consensus_queue.cc:1035] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:43.540130 6002 consensus_queue.cc:1035] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
W20240501 20:09:43.779501 4434 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.65:42941, user_credentials={real_user=slave}} blocked reactor thread for 112204us
INSERT report
rows total: 2000
time total: 317.725 ms
time per row: 0.158862 ms
I20240501 20:09:43.987746 4085 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:43.996635 4346 tablet_service.cc:1467] Processing CreateTablet for tablet 0b14eb56de3940de9fbf03d31b860005 (DEFAULT_TABLE table=rebalance_test_table_10 [id=ece9130016fb4578a855bb117efb9bca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:43.996770 4611 tablet_service.cc:1467] Processing CreateTablet for tablet 0b14eb56de3940de9fbf03d31b860005 (DEFAULT_TABLE table=rebalance_test_table_10 [id=ece9130016fb4578a855bb117efb9bca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:43.997114 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b14eb56de3940de9fbf03d31b860005. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:43.997217 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b14eb56de3940de9fbf03d31b860005. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.001227 4609 tablet_service.cc:1467] Processing CreateTablet for tablet 8afb9c6ed6f946eeb7a015a0f6f0c658 (DEFAULT_TABLE table=rebalance_test_table_10 [id=ece9130016fb4578a855bb117efb9bca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:44.001284 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 8afb9c6ed6f946eeb7a015a0f6f0c658 (DEFAULT_TABLE table=rebalance_test_table_10 [id=ece9130016fb4578a855bb117efb9bca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:44.001514 4347 tablet_service.cc:1467] Processing CreateTablet for tablet 11c38f7be4804540806d743e1fa47b48 (DEFAULT_TABLE table=rebalance_test_table_10 [id=ece9130016fb4578a855bb117efb9bca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:44.001713 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8afb9c6ed6f946eeb7a015a0f6f0c658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.001726 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8afb9c6ed6f946eeb7a015a0f6f0c658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.001931 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c38f7be4804540806d743e1fa47b48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.007071 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 11c38f7be4804540806d743e1fa47b48 (DEFAULT_TABLE table=rebalance_test_table_10 [id=ece9130016fb4578a855bb117efb9bca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:44.007071 4213 tablet_service.cc:1467] Processing CreateTablet for tablet 0b14eb56de3940de9fbf03d31b860005 (DEFAULT_TABLE table=rebalance_test_table_10 [id=ece9130016fb4578a855bb117efb9bca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:44.007481 4213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b14eb56de3940de9fbf03d31b860005. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.007481 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c38f7be4804540806d743e1fa47b48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.011168 4479 tablet_service.cc:1467] Processing CreateTablet for tablet 8afb9c6ed6f946eeb7a015a0f6f0c658 (DEFAULT_TABLE table=rebalance_test_table_10 [id=ece9130016fb4578a855bb117efb9bca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:44.011456 4214 tablet_service.cc:1467] Processing CreateTablet for tablet 11c38f7be4804540806d743e1fa47b48 (DEFAULT_TABLE table=rebalance_test_table_10 [id=ece9130016fb4578a855bb117efb9bca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:44.011629 4479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8afb9c6ed6f946eeb7a015a0f6f0c658. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.018456 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c38f7be4804540806d743e1fa47b48. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:44.019184 6115 tablet_bootstrap.cc:492] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:44.022223 6115 tablet_bootstrap.cc:654] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:44.022575 6116 tablet_bootstrap.cc:492] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:44.030400 6116 tablet_bootstrap.cc:654] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:44.036310 6117 tablet_bootstrap.cc:492] T 11c38f7be4804540806d743e1fa47b48 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:44.037487 6116 tablet_bootstrap.cc:492] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:44.037715 6116 ts_tablet_manager.cc:1367] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240501 20:09:44.038317 6117 tablet_bootstrap.cc:654] T 11c38f7be4804540806d743e1fa47b48 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:44.038581 6116 raft_consensus.cc:348] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:44.038938 6116 raft_consensus.cc:374] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:44.039106 6116 raft_consensus.cc:729] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:44.039434 6116 consensus_queue.cc:260] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:44.039870 6114 tablet_bootstrap.cc:492] T 0b14eb56de3940de9fbf03d31b860005 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:44.040858 6116 ts_tablet_manager.cc:1398] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:44.041066 6115 tablet_bootstrap.cc:492] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:44.041224 6115 ts_tablet_manager.cc:1367] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.017s
I20240501 20:09:44.041800 6114 tablet_bootstrap.cc:654] T 0b14eb56de3940de9fbf03d31b860005 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:44.042026 6117 tablet_bootstrap.cc:492] T 11c38f7be4804540806d743e1fa47b48 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:44.042066 6115 raft_consensus.cc:348] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:44.042213 6117 ts_tablet_manager.cc:1367] T 11c38f7be4804540806d743e1fa47b48 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:09:44.042289 6115 raft_consensus.cc:374] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:44.042384 6115 raft_consensus.cc:729] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:44.042680 6115 consensus_queue.cc:260] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:44.043241 6117 raft_consensus.cc:348] T 11c38f7be4804540806d743e1fa47b48 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:44.043473 6117 raft_consensus.cc:374] T 11c38f7be4804540806d743e1fa47b48 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:44.043594 6117 raft_consensus.cc:729] T 11c38f7be4804540806d743e1fa47b48 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:44.043853 6117 consensus_queue.cc:260] T 11c38f7be4804540806d743e1fa47b48 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:44.044507 6117 ts_tablet_manager.cc:1398] T 11c38f7be4804540806d743e1fa47b48 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:44.044862 6117 tablet_bootstrap.cc:492] T 0b14eb56de3940de9fbf03d31b860005 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:44.046681 6117 tablet_bootstrap.cc:654] T 0b14eb56de3940de9fbf03d31b860005 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:44.047294 6115 ts_tablet_manager.cc:1398] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:09:44.048666 6115 tablet_bootstrap.cc:492] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:44.050410 6117 tablet_bootstrap.cc:492] T 0b14eb56de3940de9fbf03d31b860005 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:44.050652 6117 ts_tablet_manager.cc:1367] T 0b14eb56de3940de9fbf03d31b860005 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240501 20:09:44.050801 6115 tablet_bootstrap.cc:654] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:44.051555 6117 raft_consensus.cc:348] T 0b14eb56de3940de9fbf03d31b860005 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:44.051873 6117 raft_consensus.cc:374] T 0b14eb56de3940de9fbf03d31b860005 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:44.052055 6117 raft_consensus.cc:729] T 0b14eb56de3940de9fbf03d31b860005 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:44.052376 6117 consensus_queue.cc:260] T 0b14eb56de3940de9fbf03d31b860005 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:44.053025 6117 ts_tablet_manager.cc:1398] T 0b14eb56de3940de9fbf03d31b860005 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:44.054342 6114 tablet_bootstrap.cc:492] T 0b14eb56de3940de9fbf03d31b860005 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:44.054414 6115 tablet_bootstrap.cc:492] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:44.054507 6114 ts_tablet_manager.cc:1367] T 0b14eb56de3940de9fbf03d31b860005 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20240501 20:09:44.054610 6115 ts_tablet_manager.cc:1367] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20240501 20:09:44.055348 6114 raft_consensus.cc:348] T 0b14eb56de3940de9fbf03d31b860005 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:44.055555 6114 raft_consensus.cc:374] T 0b14eb56de3940de9fbf03d31b860005 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:44.055476 6115 raft_consensus.cc:348] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:44.055671 6115 raft_consensus.cc:374] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:44.055684 6114 raft_consensus.cc:729] T 0b14eb56de3940de9fbf03d31b860005 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:44.055800 6115 raft_consensus.cc:729] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:44.055953 6114 consensus_queue.cc:260] T 0b14eb56de3940de9fbf03d31b860005 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:44.056078 6115 consensus_queue.cc:260] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:44.056653 6115 ts_tablet_manager.cc:1398] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:09:44.056939 6115 tablet_bootstrap.cc:492] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:44.057487 6114 ts_tablet_manager.cc:1398] T 0b14eb56de3940de9fbf03d31b860005 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:44.057792 6114 tablet_bootstrap.cc:492] T 11c38f7be4804540806d743e1fa47b48 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:44.058590 6115 tablet_bootstrap.cc:654] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:44.059634 6114 tablet_bootstrap.cc:654] T 11c38f7be4804540806d743e1fa47b48 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:44.061961 6115 tablet_bootstrap.cc:492] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:44.062111 6115 ts_tablet_manager.cc:1367] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:09:44.063160 6115 raft_consensus.cc:348] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:44.063326 6115 raft_consensus.cc:374] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:44.063423 6115 raft_consensus.cc:729] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:44.063670 6114 tablet_bootstrap.cc:492] T 11c38f7be4804540806d743e1fa47b48 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:44.063619 6115 consensus_queue.cc:260] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:44.063810 6114 ts_tablet_manager.cc:1367] T 11c38f7be4804540806d743e1fa47b48 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:09:44.064512 6114 raft_consensus.cc:348] T 11c38f7be4804540806d743e1fa47b48 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:44.064738 6114 raft_consensus.cc:374] T 11c38f7be4804540806d743e1fa47b48 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:44.064821 6115 ts_tablet_manager.cc:1398] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:44.064824 6114 raft_consensus.cc:729] T 11c38f7be4804540806d743e1fa47b48 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:44.065120 6114 consensus_queue.cc:260] T 11c38f7be4804540806d743e1fa47b48 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:44.065618 6114 ts_tablet_manager.cc:1398] T 11c38f7be4804540806d743e1fa47b48 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:44.066058 6114 tablet_bootstrap.cc:492] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:44.067683 6114 tablet_bootstrap.cc:654] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:44.069869 6114 tablet_bootstrap.cc:492] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:44.070006 6114 ts_tablet_manager.cc:1367] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 20:09:44.070798 6114 raft_consensus.cc:348] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:44.071197 6114 raft_consensus.cc:374] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:44.071298 6114 raft_consensus.cc:729] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:44.071499 6114 consensus_queue.cc:260] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:44.072026 6114 ts_tablet_manager.cc:1398] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:44.079545 6100 raft_consensus.cc:482] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:44.079715 6100 raft_consensus.cc:504] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:44.080612 6100 leader_election.cc:288] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:44.081038 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c38f7be4804540806d743e1fa47b48" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:44.081100 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c38f7be4804540806d743e1fa47b48" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:44.081259 4368 raft_consensus.cc:2454] T 11c38f7be4804540806d743e1fa47b48 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:44.081279 4631 raft_consensus.cc:2454] T 11c38f7be4804540806d743e1fa47b48 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:44.081683 4169 leader_election.cc:302] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:44.081935 6100 raft_consensus.cc:2781] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:44.082046 6100 raft_consensus.cc:482] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:44.082123 6100 raft_consensus.cc:3037] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:44.084159 6100 raft_consensus.cc:504] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:44.084818 6100 leader_election.cc:288] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:44.085209 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c38f7be4804540806d743e1fa47b48" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:44.085398 4368 raft_consensus.cc:3037] T 11c38f7be4804540806d743e1fa47b48 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:44.085343 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c38f7be4804540806d743e1fa47b48" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:44.085525 4631 raft_consensus.cc:3037] T 11c38f7be4804540806d743e1fa47b48 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:44.087534 4368 raft_consensus.cc:2454] T 11c38f7be4804540806d743e1fa47b48 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:44.087550 4631 raft_consensus.cc:2454] T 11c38f7be4804540806d743e1fa47b48 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:44.087910 4169 leader_election.cc:302] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:44.088227 5857 raft_consensus.cc:2781] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:44.090829 5857 raft_consensus.cc:686] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:09:44.091079 5857 consensus_queue.cc:237] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:44.094014 4090 catalog_manager.cc:5484] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:44.239007 5962 raft_consensus.cc:482] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:44.239195 5962 raft_consensus.cc:504] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:44.239943 5962 leader_election.cc:288] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:09:44.240442 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8afb9c6ed6f946eeb7a015a0f6f0c658" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:44.240532 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8afb9c6ed6f946eeb7a015a0f6f0c658" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:44.240669 4498 raft_consensus.cc:2454] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:09:44.240756 4235 raft_consensus.cc:2454] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:09:44.241190 4564 leader_election.cc:302] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:44.241494 5962 raft_consensus.cc:2781] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:44.241644 5962 raft_consensus.cc:482] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:44.241791 5962 raft_consensus.cc:3037] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:44.244138 5962 raft_consensus.cc:504] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:44.244850 5962 leader_election.cc:288] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:09:44.245332 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8afb9c6ed6f946eeb7a015a0f6f0c658" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:44.245363 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8afb9c6ed6f946eeb7a015a0f6f0c658" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:44.245544 4498 raft_consensus.cc:3037] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:44.245596 4233 raft_consensus.cc:3037] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:44.247471 4498 raft_consensus.cc:2454] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:09:44.247910 4233 raft_consensus.cc:2454] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:09:44.247895 4564 leader_election.cc:302] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:44.248252 5962 raft_consensus.cc:2781] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:44.248734 5962 raft_consensus.cc:686] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Becoming Leader. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Running, Role: LEADER
I20240501 20:09:44.249061 5962 consensus_queue.cc:237] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:44.252384 4090 catalog_manager.cc:5484] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68). New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:44.296177 5857 raft_consensus.cc:482] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:44.296402 5857 raft_consensus.cc:504] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:44.297122 5857 leader_election.cc:288] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:44.297616 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b14eb56de3940de9fbf03d31b860005" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:44.297677 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b14eb56de3940de9fbf03d31b860005" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:44.297932 4631 raft_consensus.cc:2454] T 0b14eb56de3940de9fbf03d31b860005 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:44.297932 4368 raft_consensus.cc:2454] T 0b14eb56de3940de9fbf03d31b860005 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:44.298344 4170 leader_election.cc:302] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:44.298740 5857 raft_consensus.cc:2781] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:44.298853 5857 raft_consensus.cc:482] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:44.298996 5857 raft_consensus.cc:3037] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:44.301065 5857 raft_consensus.cc:504] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:44.301715 5857 leader_election.cc:288] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:44.301977 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b14eb56de3940de9fbf03d31b860005" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:44.302210 4368 raft_consensus.cc:3037] T 0b14eb56de3940de9fbf03d31b860005 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:44.302163 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b14eb56de3940de9fbf03d31b860005" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:44.302383 4631 raft_consensus.cc:3037] T 0b14eb56de3940de9fbf03d31b860005 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:44.305121 4368 raft_consensus.cc:2454] T 0b14eb56de3940de9fbf03d31b860005 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:44.305131 4631 raft_consensus.cc:2454] T 0b14eb56de3940de9fbf03d31b860005 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:44.305510 4169 leader_election.cc:302] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:44.305821 5857 raft_consensus.cc:2781] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:44.306021 5857 raft_consensus.cc:686] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:09:44.306280 5857 consensus_queue.cc:237] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:44.309172 4090 catalog_manager.cc:5484] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:44.491827 5857 consensus_queue.cc:1035] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:44.497267 6127 consensus_queue.cc:1035] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:44.664180 5962 consensus_queue.cc:1035] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:44.671422 6060 consensus_queue.cc:1035] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:44.676590 6132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:44.677053 6132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:44.685324 6132 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240501 20:09:44.795958 6127 consensus_queue.cc:1035] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:44.805094 6127 consensus_queue.cc:1035] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 252.979 ms
time per row: 0.126489 ms
I20240501 20:09:45.096587 4087 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:45.105845 4479 tablet_service.cc:1467] Processing CreateTablet for tablet 34cb2f3c90cc4158a61f426c0e5cb86b (DEFAULT_TABLE table=rebalance_test_table_11 [id=3b02a4c6c0f54681a87f5ca2381de01d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:45.105865 4213 tablet_service.cc:1467] Processing CreateTablet for tablet 34cb2f3c90cc4158a61f426c0e5cb86b (DEFAULT_TABLE table=rebalance_test_table_11 [id=3b02a4c6c0f54681a87f5ca2381de01d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:45.106317 4213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34cb2f3c90cc4158a61f426c0e5cb86b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.106320 4479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34cb2f3c90cc4158a61f426c0e5cb86b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.106496 4611 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6c89fa7e6f485e889a5197da4bd135 (DEFAULT_TABLE table=rebalance_test_table_11 [id=3b02a4c6c0f54681a87f5ca2381de01d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:45.107012 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6c89fa7e6f485e889a5197da4bd135. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.111900 4609 tablet_service.cc:1467] Processing CreateTablet for tablet c66514deec5744da832bc030a582c46a (DEFAULT_TABLE table=rebalance_test_table_11 [id=3b02a4c6c0f54681a87f5ca2381de01d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:45.112344 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c66514deec5744da832bc030a582c46a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.114236 4347 tablet_service.cc:1467] Processing CreateTablet for tablet 34cb2f3c90cc4158a61f426c0e5cb86b (DEFAULT_TABLE table=rebalance_test_table_11 [id=3b02a4c6c0f54681a87f5ca2381de01d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:45.114667 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34cb2f3c90cc4158a61f426c0e5cb86b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.115345 6192 tablet_bootstrap.cc:492] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:45.117194 6192 tablet_bootstrap.cc:654] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:45.119150 4346 tablet_service.cc:1467] Processing CreateTablet for tablet c66514deec5744da832bc030a582c46a (DEFAULT_TABLE table=rebalance_test_table_11 [id=3b02a4c6c0f54681a87f5ca2381de01d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:45.119647 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c66514deec5744da832bc030a582c46a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.121549 6193 tablet_bootstrap.cc:492] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:45.122905 4477 tablet_service.cc:1467] Processing CreateTablet for tablet c66514deec5744da832bc030a582c46a (DEFAULT_TABLE table=rebalance_test_table_11 [id=3b02a4c6c0f54681a87f5ca2381de01d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:45.123051 4213 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6c89fa7e6f485e889a5197da4bd135 (DEFAULT_TABLE table=rebalance_test_table_11 [id=3b02a4c6c0f54681a87f5ca2381de01d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:45.123317 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c66514deec5744da832bc030a582c46a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.123467 4213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6c89fa7e6f485e889a5197da4bd135. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.123553 6193 tablet_bootstrap.cc:654] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:45.126045 6192 tablet_bootstrap.cc:492] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:45.126251 6192 ts_tablet_manager.cc:1367] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.001s
I20240501 20:09:45.127198 4478 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6c89fa7e6f485e889a5197da4bd135 (DEFAULT_TABLE table=rebalance_test_table_11 [id=3b02a4c6c0f54681a87f5ca2381de01d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:45.127314 6192 raft_consensus.cc:348] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:45.127580 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6c89fa7e6f485e889a5197da4bd135. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:45.127580 6192 raft_consensus.cc:374] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:45.127681 6192 raft_consensus.cc:729] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:45.130872 6192 consensus_queue.cc:260] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:45.131958 6192 ts_tablet_manager.cc:1398] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240501 20:09:45.132344 6192 tablet_bootstrap.cc:492] T c66514deec5744da832bc030a582c46a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:45.134531 6192 tablet_bootstrap.cc:654] T c66514deec5744da832bc030a582c46a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:45.148622 6196 tablet_bootstrap.cc:492] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:45.151129 6196 tablet_bootstrap.cc:654] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:45.152864 6197 tablet_bootstrap.cc:492] T 34cb2f3c90cc4158a61f426c0e5cb86b P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:45.153075 6193 tablet_bootstrap.cc:492] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:45.153139 6192 tablet_bootstrap.cc:492] T c66514deec5744da832bc030a582c46a P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:45.153231 6193 ts_tablet_manager.cc:1367] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.032s user 0.015s sys 0.015s
I20240501 20:09:45.153312 6192 ts_tablet_manager.cc:1367] T c66514deec5744da832bc030a582c46a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.004s
I20240501 20:09:45.153882 6193 raft_consensus.cc:348] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:45.154124 6193 raft_consensus.cc:374] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:45.154215 6193 raft_consensus.cc:729] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:45.154172 6192 raft_consensus.cc:348] T c66514deec5744da832bc030a582c46a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:45.154381 6192 raft_consensus.cc:374] T c66514deec5744da832bc030a582c46a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:45.154453 6192 raft_consensus.cc:729] T c66514deec5744da832bc030a582c46a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:45.154408 6193 consensus_queue.cc:260] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:45.154853 6197 tablet_bootstrap.cc:654] T 34cb2f3c90cc4158a61f426c0e5cb86b P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:45.155022 6193 ts_tablet_manager.cc:1398] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:45.154904 6192 consensus_queue.cc:260] T c66514deec5744da832bc030a582c46a P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:45.155345 6193 tablet_bootstrap.cc:492] T 1c6c89fa7e6f485e889a5197da4bd135 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:45.155743 6192 ts_tablet_manager.cc:1398] T c66514deec5744da832bc030a582c46a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 20:09:45.156939 6193 tablet_bootstrap.cc:654] T 1c6c89fa7e6f485e889a5197da4bd135 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:45.157181 6196 tablet_bootstrap.cc:492] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:45.157346 6196 ts_tablet_manager.cc:1367] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240501 20:09:45.158213 6196 raft_consensus.cc:348] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:45.158565 6196 raft_consensus.cc:374] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:45.158758 6196 raft_consensus.cc:729] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:45.159152 6196 consensus_queue.cc:260] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:45.162387 6196 ts_tablet_manager.cc:1398] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:09:45.165568 6196 tablet_bootstrap.cc:492] T 34cb2f3c90cc4158a61f426c0e5cb86b P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:45.165828 6193 tablet_bootstrap.cc:492] T 1c6c89fa7e6f485e889a5197da4bd135 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:45.166011 6193 ts_tablet_manager.cc:1367] T 1c6c89fa7e6f485e889a5197da4bd135 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.006s
I20240501 20:09:45.166976 6193 raft_consensus.cc:348] T 1c6c89fa7e6f485e889a5197da4bd135 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:45.167179 6193 raft_consensus.cc:374] T 1c6c89fa7e6f485e889a5197da4bd135 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:45.167261 6193 raft_consensus.cc:729] T 1c6c89fa7e6f485e889a5197da4bd135 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:45.167506 6193 consensus_queue.cc:260] T 1c6c89fa7e6f485e889a5197da4bd135 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:45.168134 6193 ts_tablet_manager.cc:1398] T 1c6c89fa7e6f485e889a5197da4bd135 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:45.169631 6197 tablet_bootstrap.cc:492] T 34cb2f3c90cc4158a61f426c0e5cb86b P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:45.169790 6197 ts_tablet_manager.cc:1367] T 34cb2f3c90cc4158a61f426c0e5cb86b P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.010s
I20240501 20:09:45.170609 6197 raft_consensus.cc:348] T 34cb2f3c90cc4158a61f426c0e5cb86b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:45.171103 6196 tablet_bootstrap.cc:654] T 34cb2f3c90cc4158a61f426c0e5cb86b P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:45.170854 6197 raft_consensus.cc:374] T 34cb2f3c90cc4158a61f426c0e5cb86b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:45.171397 6197 raft_consensus.cc:729] T 34cb2f3c90cc4158a61f426c0e5cb86b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:45.171663 6197 consensus_queue.cc:260] T 34cb2f3c90cc4158a61f426c0e5cb86b P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:45.172410 6197 ts_tablet_manager.cc:1398] T 34cb2f3c90cc4158a61f426c0e5cb86b P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 20:09:45.172775 6197 tablet_bootstrap.cc:492] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:45.175006 6197 tablet_bootstrap.cc:654] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:45.177878 6196 tablet_bootstrap.cc:492] T 34cb2f3c90cc4158a61f426c0e5cb86b P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:45.178135 6196 ts_tablet_manager.cc:1367] T 34cb2f3c90cc4158a61f426c0e5cb86b P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.003s
I20240501 20:09:45.178268 6197 tablet_bootstrap.cc:492] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:45.178457 6197 ts_tablet_manager.cc:1367] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240501 20:09:45.179076 6196 raft_consensus.cc:348] T 34cb2f3c90cc4158a61f426c0e5cb86b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:45.179370 6196 raft_consensus.cc:374] T 34cb2f3c90cc4158a61f426c0e5cb86b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:45.179315 6197 raft_consensus.cc:348] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:45.179553 6197 raft_consensus.cc:374] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:45.179545 6196 raft_consensus.cc:729] T 34cb2f3c90cc4158a61f426c0e5cb86b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:45.179649 6197 raft_consensus.cc:729] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:45.179903 6196 consensus_queue.cc:260] T 34cb2f3c90cc4158a61f426c0e5cb86b P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:45.179911 6197 consensus_queue.cc:260] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:45.180509 6196 ts_tablet_manager.cc:1398] T 34cb2f3c90cc4158a61f426c0e5cb86b P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:45.180507 6197 ts_tablet_manager.cc:1398] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:45.180970 6196 tablet_bootstrap.cc:492] T 1c6c89fa7e6f485e889a5197da4bd135 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:45.182752 6196 tablet_bootstrap.cc:654] T 1c6c89fa7e6f485e889a5197da4bd135 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:45.184828 6196 tablet_bootstrap.cc:492] T 1c6c89fa7e6f485e889a5197da4bd135 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:45.185031 6196 ts_tablet_manager.cc:1367] T 1c6c89fa7e6f485e889a5197da4bd135 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20240501 20:09:45.185950 6196 raft_consensus.cc:348] T 1c6c89fa7e6f485e889a5197da4bd135 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:45.186240 6196 raft_consensus.cc:374] T 1c6c89fa7e6f485e889a5197da4bd135 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:45.186376 6196 raft_consensus.cc:729] T 1c6c89fa7e6f485e889a5197da4bd135 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:45.186723 6196 consensus_queue.cc:260] T 1c6c89fa7e6f485e889a5197da4bd135 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:45.187286 6196 ts_tablet_manager.cc:1398] T 1c6c89fa7e6f485e889a5197da4bd135 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:45.201267 6184 raft_consensus.cc:482] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:45.201478 6184 raft_consensus.cc:504] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:45.202180 6184 leader_election.cc:288] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:45.202669 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cb2f3c90cc4158a61f426c0e5cb86b" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:45.202797 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cb2f3c90cc4158a61f426c0e5cb86b" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:45.202905 4498 raft_consensus.cc:2454] T 34cb2f3c90cc4158a61f426c0e5cb86b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:45.203022 4368 raft_consensus.cc:2454] T 34cb2f3c90cc4158a61f426c0e5cb86b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:45.203346 4169 leader_election.cc:302] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:45.203742 6184 raft_consensus.cc:2781] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:45.203876 6184 raft_consensus.cc:482] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:45.203972 6184 raft_consensus.cc:3037] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:45.205860 6184 raft_consensus.cc:504] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:45.206514 6184 leader_election.cc:288] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:45.207043 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cb2f3c90cc4158a61f426c0e5cb86b" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:45.207253 4498 raft_consensus.cc:3037] T 34cb2f3c90cc4158a61f426c0e5cb86b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:45.207168 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34cb2f3c90cc4158a61f426c0e5cb86b" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:45.207397 4368 raft_consensus.cc:3037] T 34cb2f3c90cc4158a61f426c0e5cb86b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:45.209836 4498 raft_consensus.cc:2454] T 34cb2f3c90cc4158a61f426c0e5cb86b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:45.209919 4368 raft_consensus.cc:2454] T 34cb2f3c90cc4158a61f426c0e5cb86b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:45.210394 4169 leader_election.cc:302] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:45.210685 6184 raft_consensus.cc:2781] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:45.210840 6184 raft_consensus.cc:686] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:09:45.211100 6184 consensus_queue.cc:237] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:45.214360 4087 catalog_manager.cc:5484] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:45.255514 6088 raft_consensus.cc:482] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:45.255730 6088 raft_consensus.cc:504] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:45.256601 6088 leader_election.cc:288] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:45.257186 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66514deec5744da832bc030a582c46a" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:45.257186 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66514deec5744da832bc030a582c46a" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:45.257460 4498 raft_consensus.cc:2454] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:09:45.257459 4630 raft_consensus.cc:2454] T c66514deec5744da832bc030a582c46a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:09:45.257982 4302 leader_election.cc:302] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 1d34e18a1d394d3f91c9f50d1c7bea4e; no voters:
I20240501 20:09:45.258288 6088 raft_consensus.cc:2781] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:45.258407 6088 raft_consensus.cc:482] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:45.258489 6088 raft_consensus.cc:3037] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:45.260433 6088 raft_consensus.cc:504] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:45.260973 6088 leader_election.cc:288] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:45.261538 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66514deec5744da832bc030a582c46a" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:45.261577 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66514deec5744da832bc030a582c46a" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:45.261750 4630 raft_consensus.cc:3037] T c66514deec5744da832bc030a582c46a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:45.261862 4498 raft_consensus.cc:3037] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:45.264225 4498 raft_consensus.cc:2454] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:09:45.264456 4630 raft_consensus.cc:2454] T c66514deec5744da832bc030a582c46a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:09:45.264657 4301 leader_election.cc:302] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:45.266590 6088 raft_consensus.cc:2781] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:45.267083 6088 raft_consensus.cc:686] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Becoming Leader. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Running, Role: LEADER
I20240501 20:09:45.267426 6088 consensus_queue.cc:237] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:45.271132 4087 catalog_manager.cc:5484] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a reported cstate change: term changed from 0 to 1, leader changed from <none> to 10b47782558c41ee91f5e164a372049a (127.29.219.66). New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:45.290431 5962 raft_consensus.cc:482] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:45.290671 5962 raft_consensus.cc:504] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:45.291608 5962 leader_election.cc:288] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:09:45.292145 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6c89fa7e6f485e889a5197da4bd135" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:45.292385 4498 raft_consensus.cc:2454] T 1c6c89fa7e6f485e889a5197da4bd135 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:09:45.292382 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6c89fa7e6f485e889a5197da4bd135" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:45.292580 4235 raft_consensus.cc:2454] T 1c6c89fa7e6f485e889a5197da4bd135 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:09:45.292848 4564 leader_election.cc:302] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:45.293144 5962 raft_consensus.cc:2781] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:45.293256 5962 raft_consensus.cc:482] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:45.293337 5962 raft_consensus.cc:3037] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:45.296186 5962 raft_consensus.cc:504] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:45.296802 5962 leader_election.cc:288] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:09:45.297247 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6c89fa7e6f485e889a5197da4bd135" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:45.297315 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6c89fa7e6f485e889a5197da4bd135" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:45.297466 4498 raft_consensus.cc:3037] T 1c6c89fa7e6f485e889a5197da4bd135 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:45.297516 4235 raft_consensus.cc:3037] T 1c6c89fa7e6f485e889a5197da4bd135 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:45.300364 4498 raft_consensus.cc:2454] T 1c6c89fa7e6f485e889a5197da4bd135 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:09:45.300449 4235 raft_consensus.cc:2454] T 1c6c89fa7e6f485e889a5197da4bd135 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:09:45.300858 4564 leader_election.cc:302] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:45.301155 5962 raft_consensus.cc:2781] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:45.301286 5962 raft_consensus.cc:686] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Becoming Leader. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Running, Role: LEADER
I20240501 20:09:45.301579 5962 consensus_queue.cc:237] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:45.304656 4087 catalog_manager.cc:5484] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68). New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:45.660645 6184 consensus_queue.cc:1035] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:45.660653 6088 consensus_queue.cc:1035] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:45.672159 5857 consensus_queue.cc:1035] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:45.689713 6175 consensus_queue.cc:1035] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:45.692687 6088 consensus_queue.cc:1035] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:45.743376 5962 consensus_queue.cc:1035] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:45.744463 6209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:45.744904 6209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:45.755694 6209 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
INSERT report
rows total: 2000
time total: 272.923 ms
time per row: 0.136462 ms
I20240501 20:09:46.186733 4093 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:46.196709 4347 tablet_service.cc:1467] Processing CreateTablet for tablet 1d03fbe032b344d98d8f500c2667dd44 (DEFAULT_TABLE table=rebalance_test_table_12 [id=c3c3c4ff4c8444a38dae5dc10a57d879]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:46.197206 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d03fbe032b344d98d8f500c2667dd44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:46.198123 4609 tablet_service.cc:1467] Processing CreateTablet for tablet c78926d6cd5d44c6a19ee07396984b61 (DEFAULT_TABLE table=rebalance_test_table_12 [id=c3c3c4ff4c8444a38dae5dc10a57d879]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:46.198185 4611 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf9f8a78da54553bfe6eaa1f3b28a3f (DEFAULT_TABLE table=rebalance_test_table_12 [id=c3c3c4ff4c8444a38dae5dc10a57d879]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:46.198770 4477 tablet_service.cc:1467] Processing CreateTablet for tablet 1d03fbe032b344d98d8f500c2667dd44 (DEFAULT_TABLE table=rebalance_test_table_12 [id=c3c3c4ff4c8444a38dae5dc10a57d879]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:46.199167 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d03fbe032b344d98d8f500c2667dd44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:46.198590 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c78926d6cd5d44c6a19ee07396984b61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:46.199450 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf9f8a78da54553bfe6eaa1f3b28a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:46.203017 4478 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf9f8a78da54553bfe6eaa1f3b28a3f (DEFAULT_TABLE table=rebalance_test_table_12 [id=c3c3c4ff4c8444a38dae5dc10a57d879]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:46.203413 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf9f8a78da54553bfe6eaa1f3b28a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:46.206277 6281 tablet_bootstrap.cc:492] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:46.207454 4214 tablet_service.cc:1467] Processing CreateTablet for tablet c78926d6cd5d44c6a19ee07396984b61 (DEFAULT_TABLE table=rebalance_test_table_12 [id=c3c3c4ff4c8444a38dae5dc10a57d879]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:46.207927 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c78926d6cd5d44c6a19ee07396984b61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:46.208168 4347 tablet_service.cc:1467] Processing CreateTablet for tablet c78926d6cd5d44c6a19ee07396984b61 (DEFAULT_TABLE table=rebalance_test_table_12 [id=c3c3c4ff4c8444a38dae5dc10a57d879]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:46.208670 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c78926d6cd5d44c6a19ee07396984b61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:46.211345 4213 tablet_service.cc:1467] Processing CreateTablet for tablet 1d03fbe032b344d98d8f500c2667dd44 (DEFAULT_TABLE table=rebalance_test_table_12 [id=c3c3c4ff4c8444a38dae5dc10a57d879]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:46.211755 4213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d03fbe032b344d98d8f500c2667dd44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:46.207500 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf9f8a78da54553bfe6eaa1f3b28a3f (DEFAULT_TABLE table=rebalance_test_table_12 [id=c3c3c4ff4c8444a38dae5dc10a57d879]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:46.215356 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf9f8a78da54553bfe6eaa1f3b28a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:46.222205 6283 tablet_bootstrap.cc:492] T 1d03fbe032b344d98d8f500c2667dd44 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:46.224215 6283 tablet_bootstrap.cc:654] T 1d03fbe032b344d98d8f500c2667dd44 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:46.225126 6282 tablet_bootstrap.cc:492] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:46.227186 6282 tablet_bootstrap.cc:654] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:46.235483 6282 tablet_bootstrap.cc:492] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:46.235741 6282 ts_tablet_manager.cc:1367] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.001s
I20240501 20:09:46.236747 6282 raft_consensus.cc:348] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:46.237670 6282 raft_consensus.cc:374] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:46.237838 6282 raft_consensus.cc:729] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:46.238166 6282 consensus_queue.cc:260] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:46.239065 6282 ts_tablet_manager.cc:1398] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:46.240824 6281 tablet_bootstrap.cc:654] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:46.241160 6282 tablet_bootstrap.cc:492] T c78926d6cd5d44c6a19ee07396984b61 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:46.243077 6282 tablet_bootstrap.cc:654] T c78926d6cd5d44c6a19ee07396984b61 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:46.246234 6286 tablet_bootstrap.cc:492] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:46.247363 6282 tablet_bootstrap.cc:492] T c78926d6cd5d44c6a19ee07396984b61 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:46.247527 6282 ts_tablet_manager.cc:1367] T c78926d6cd5d44c6a19ee07396984b61 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 20:09:46.248164 6286 tablet_bootstrap.cc:654] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:46.248219 6282 raft_consensus.cc:348] T c78926d6cd5d44c6a19ee07396984b61 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:46.248476 6282 raft_consensus.cc:374] T c78926d6cd5d44c6a19ee07396984b61 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:46.248559 6282 raft_consensus.cc:729] T c78926d6cd5d44c6a19ee07396984b61 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:46.248809 6282 consensus_queue.cc:260] T c78926d6cd5d44c6a19ee07396984b61 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:46.249372 6282 ts_tablet_manager.cc:1398] T c78926d6cd5d44c6a19ee07396984b61 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:46.250651 6281 tablet_bootstrap.cc:492] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:46.251140 6286 tablet_bootstrap.cc:492] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:46.251295 6281 ts_tablet_manager.cc:1367] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.045s user 0.001s sys 0.003s
I20240501 20:09:46.251309 6286 ts_tablet_manager.cc:1367] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:09:46.251869 6283 tablet_bootstrap.cc:492] T 1d03fbe032b344d98d8f500c2667dd44 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:46.252046 6283 ts_tablet_manager.cc:1367] T 1d03fbe032b344d98d8f500c2667dd44 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.012s
I20240501 20:09:46.252151 6286 raft_consensus.cc:348] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:46.252395 6286 raft_consensus.cc:374] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:46.252238 6281 raft_consensus.cc:348] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:46.252521 6286 raft_consensus.cc:729] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:46.252521 6281 raft_consensus.cc:374] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:46.252835 6283 raft_consensus.cc:348] T 1d03fbe032b344d98d8f500c2667dd44 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:46.252823 6286 consensus_queue.cc:260] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:46.253043 6283 raft_consensus.cc:374] T 1d03fbe032b344d98d8f500c2667dd44 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:46.252990 6281 raft_consensus.cc:729] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:46.253146 6283 raft_consensus.cc:729] T 1d03fbe032b344d98d8f500c2667dd44 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:46.253412 6286 ts_tablet_manager.cc:1398] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:46.253394 6283 consensus_queue.cc:260] T 1d03fbe032b344d98d8f500c2667dd44 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:46.253449 6281 consensus_queue.cc:260] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:46.254513 6281 ts_tablet_manager.cc:1398] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:46.254835 6286 tablet_bootstrap.cc:492] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:46.254935 6281 tablet_bootstrap.cc:492] T c78926d6cd5d44c6a19ee07396984b61 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:46.255929 6283 ts_tablet_manager.cc:1398] T 1d03fbe032b344d98d8f500c2667dd44 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240501 20:09:46.256242 6283 tablet_bootstrap.cc:492] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:46.256736 6281 tablet_bootstrap.cc:654] T c78926d6cd5d44c6a19ee07396984b61 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:46.256774 6286 tablet_bootstrap.cc:654] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:46.257825 6283 tablet_bootstrap.cc:654] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:46.268772 6286 tablet_bootstrap.cc:492] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:46.268970 6286 ts_tablet_manager.cc:1367] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20240501 20:09:46.269327 6281 tablet_bootstrap.cc:492] T c78926d6cd5d44c6a19ee07396984b61 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:46.269405 6283 tablet_bootstrap.cc:492] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:46.269574 6283 ts_tablet_manager.cc:1367] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240501 20:09:46.269598 6281 ts_tablet_manager.cc:1367] T c78926d6cd5d44c6a19ee07396984b61 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.001s
I20240501 20:09:46.270401 6283 raft_consensus.cc:348] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:46.270431 6281 raft_consensus.cc:348] T c78926d6cd5d44c6a19ee07396984b61 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:46.270669 6283 raft_consensus.cc:374] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:46.270679 6281 raft_consensus.cc:374] T c78926d6cd5d44c6a19ee07396984b61 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:46.270772 6281 raft_consensus.cc:729] T c78926d6cd5d44c6a19ee07396984b61 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:46.270769 6283 raft_consensus.cc:729] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:46.271006 6281 consensus_queue.cc:260] T c78926d6cd5d44c6a19ee07396984b61 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:46.271023 6283 consensus_queue.cc:260] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:46.271590 6283 ts_tablet_manager.cc:1398] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:46.271792 6281 ts_tablet_manager.cc:1398] T c78926d6cd5d44c6a19ee07396984b61 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:09:46.273007 6286 raft_consensus.cc:348] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:46.273248 6286 raft_consensus.cc:374] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:46.273378 6286 raft_consensus.cc:729] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:46.273665 6286 consensus_queue.cc:260] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:46.274228 6286 ts_tablet_manager.cc:1398] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 20:09:46.274497 6286 tablet_bootstrap.cc:492] T 1d03fbe032b344d98d8f500c2667dd44 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:46.276341 6286 tablet_bootstrap.cc:654] T 1d03fbe032b344d98d8f500c2667dd44 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:46.280267 6286 tablet_bootstrap.cc:492] T 1d03fbe032b344d98d8f500c2667dd44 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:46.280422 6286 ts_tablet_manager.cc:1367] T 1d03fbe032b344d98d8f500c2667dd44 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 20:09:46.281322 6286 raft_consensus.cc:348] T 1d03fbe032b344d98d8f500c2667dd44 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:46.281541 6286 raft_consensus.cc:374] T 1d03fbe032b344d98d8f500c2667dd44 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:46.281666 6286 raft_consensus.cc:729] T 1d03fbe032b344d98d8f500c2667dd44 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:46.281940 6286 consensus_queue.cc:260] T 1d03fbe032b344d98d8f500c2667dd44 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:46.282601 6286 ts_tablet_manager.cc:1398] T 1d03fbe032b344d98d8f500c2667dd44 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:46.318138 6211 raft_consensus.cc:482] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:46.318334 6211 raft_consensus.cc:504] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:46.319300 6211 leader_election.cc:288] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:46.319748 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf9f8a78da54553bfe6eaa1f3b28a3f" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:46.319878 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf9f8a78da54553bfe6eaa1f3b28a3f" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:46.319985 4235 raft_consensus.cc:2454] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:46.320115 4630 raft_consensus.cc:2454] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:46.320422 4434 leader_election.cc:302] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:46.320677 6211 raft_consensus.cc:2781] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:46.320806 6211 raft_consensus.cc:482] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:46.320897 6211 raft_consensus.cc:3037] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:46.322894 6211 raft_consensus.cc:504] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:46.323453 6211 leader_election.cc:288] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Requested vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:46.323890 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf9f8a78da54553bfe6eaa1f3b28a3f" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:46.323940 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf9f8a78da54553bfe6eaa1f3b28a3f" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:46.324133 4630 raft_consensus.cc:3037] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:46.324127 4235 raft_consensus.cc:3037] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:46.327054 4630 raft_consensus.cc:2454] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:46.327034 4235 raft_consensus.cc:2454] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:46.327523 4434 leader_election.cc:302] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:46.327929 6211 raft_consensus.cc:2781] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:46.328083 6211 raft_consensus.cc:686] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:09:46.328415 6211 consensus_queue.cc:237] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:46.328892 6261 raft_consensus.cc:482] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:46.329059 6261 raft_consensus.cc:504] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:46.329808 6261 leader_election.cc:288] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:46.330346 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78926d6cd5d44c6a19ee07396984b61" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:46.330583 4630 raft_consensus.cc:2454] T c78926d6cd5d44c6a19ee07396984b61 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:46.330540 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78926d6cd5d44c6a19ee07396984b61" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:46.330785 4365 raft_consensus.cc:2454] T c78926d6cd5d44c6a19ee07396984b61 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:46.331045 4170 leader_election.cc:302] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:46.331303 6261 raft_consensus.cc:2781] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:46.331395 6261 raft_consensus.cc:482] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:46.331481 6261 raft_consensus.cc:3037] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:46.332216 4093 catalog_manager.cc:5484] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:46.333393 6261 raft_consensus.cc:504] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:46.333997 6261 leader_election.cc:288] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:46.334776 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78926d6cd5d44c6a19ee07396984b61" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:46.334993 4365 raft_consensus.cc:3037] T c78926d6cd5d44c6a19ee07396984b61 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:46.335458 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78926d6cd5d44c6a19ee07396984b61" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:46.335687 4630 raft_consensus.cc:3037] T c78926d6cd5d44c6a19ee07396984b61 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:46.337409 4365 raft_consensus.cc:2454] T c78926d6cd5d44c6a19ee07396984b61 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:46.337909 4169 leader_election.cc:302] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:46.338096 4630 raft_consensus.cc:2454] T c78926d6cd5d44c6a19ee07396984b61 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:46.338182 6261 raft_consensus.cc:2781] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:46.338320 6261 raft_consensus.cc:686] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:09:46.338910 6261 consensus_queue.cc:237] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:46.342268 4093 catalog_manager.cc:5484] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:46.548683 6217 raft_consensus.cc:482] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:46.548879 6217 raft_consensus.cc:504] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:46.549602 6217 leader_election.cc:288] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:09:46.550149 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d03fbe032b344d98d8f500c2667dd44" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:46.550212 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d03fbe032b344d98d8f500c2667dd44" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:46.550375 4499 raft_consensus.cc:2454] T 1d03fbe032b344d98d8f500c2667dd44 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:09:46.550428 4235 raft_consensus.cc:2454] T 1d03fbe032b344d98d8f500c2667dd44 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:09:46.550787 4301 leader_election.cc:302] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:46.551103 6297 raft_consensus.cc:2781] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:46.551195 6297 raft_consensus.cc:482] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:46.551330 6297 raft_consensus.cc:3037] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:46.553311 6297 raft_consensus.cc:504] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:46.553795 6297 leader_election.cc:288] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:09:46.554247 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d03fbe032b344d98d8f500c2667dd44" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:46.554280 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d03fbe032b344d98d8f500c2667dd44" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:46.554414 4499 raft_consensus.cc:3037] T 1d03fbe032b344d98d8f500c2667dd44 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:46.554450 4235 raft_consensus.cc:3037] T 1d03fbe032b344d98d8f500c2667dd44 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:46.556586 4499 raft_consensus.cc:2454] T 1d03fbe032b344d98d8f500c2667dd44 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:09:46.556605 4235 raft_consensus.cc:2454] T 1d03fbe032b344d98d8f500c2667dd44 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:09:46.556931 4301 leader_election.cc:302] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:46.557184 6297 raft_consensus.cc:2781] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:46.557329 6297 raft_consensus.cc:686] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Becoming Leader. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Running, Role: LEADER
I20240501 20:09:46.557605 6297 consensus_queue.cc:237] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:46.560730 4093 catalog_manager.cc:5484] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a reported cstate change: term changed from 0 to 1, leader changed from <none> to 10b47782558c41ee91f5e164a372049a (127.29.219.66). New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:46.727340 6261 consensus_queue.cc:1035] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:46.748797 6265 consensus_queue.cc:1035] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:46.756753 6211 consensus_queue.cc:1035] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:46.777613 6307 consensus_queue.cc:1035] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:46.957182 6299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:46.957499 6299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:46.965898 6299 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
I20240501 20:09:47.135555 4499 raft_consensus.cc:1261] T 1d03fbe032b344d98d8f500c2667dd44 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:47.135553 4233 raft_consensus.cc:1261] T 1d03fbe032b344d98d8f500c2667dd44 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:47.136644 6297 consensus_queue.cc:1035] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:47.142450 6297 consensus_queue.cc:1035] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 274.742 ms
time per row: 0.137371 ms
I20240501 20:09:47.396173 4093 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:47.404577 4347 tablet_service.cc:1467] Processing CreateTablet for tablet 9d165c29dabe4ae3be11ed84702c3cb2 (DEFAULT_TABLE table=rebalance_test_table_13 [id=e21b0047cbf842c6833a78db81a34171]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:47.404628 4611 tablet_service.cc:1467] Processing CreateTablet for tablet 9d165c29dabe4ae3be11ed84702c3cb2 (DEFAULT_TABLE table=rebalance_test_table_13 [id=e21b0047cbf842c6833a78db81a34171]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:47.405046 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d165c29dabe4ae3be11ed84702c3cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:47.405058 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d165c29dabe4ae3be11ed84702c3cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:47.405634 4477 tablet_service.cc:1467] Processing CreateTablet for tablet 9d165c29dabe4ae3be11ed84702c3cb2 (DEFAULT_TABLE table=rebalance_test_table_13 [id=e21b0047cbf842c6833a78db81a34171]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:47.406109 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d165c29dabe4ae3be11ed84702c3cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:47.406586 4213 tablet_service.cc:1467] Processing CreateTablet for tablet 91a2bf5ddc03431b963ab5da831fd340 (DEFAULT_TABLE table=rebalance_test_table_13 [id=e21b0047cbf842c6833a78db81a34171]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:47.407021 4213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a2bf5ddc03431b963ab5da831fd340. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:47.407212 4346 tablet_service.cc:1467] Processing CreateTablet for tablet 46df0faa681b469abe9e383b181a6053 (DEFAULT_TABLE table=rebalance_test_table_13 [id=e21b0047cbf842c6833a78db81a34171]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:47.407550 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46df0faa681b469abe9e383b181a6053. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:47.410501 4609 tablet_service.cc:1467] Processing CreateTablet for tablet 46df0faa681b469abe9e383b181a6053 (DEFAULT_TABLE table=rebalance_test_table_13 [id=e21b0047cbf842c6833a78db81a34171]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:47.405674 4478 tablet_service.cc:1467] Processing CreateTablet for tablet 46df0faa681b469abe9e383b181a6053 (DEFAULT_TABLE table=rebalance_test_table_13 [id=e21b0047cbf842c6833a78db81a34171]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:47.410919 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46df0faa681b469abe9e383b181a6053. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:47.411118 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46df0faa681b469abe9e383b181a6053. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:47.417086 4348 tablet_service.cc:1467] Processing CreateTablet for tablet 91a2bf5ddc03431b963ab5da831fd340 (DEFAULT_TABLE table=rebalance_test_table_13 [id=e21b0047cbf842c6833a78db81a34171]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:47.417555 4348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a2bf5ddc03431b963ab5da831fd340. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:47.421907 4479 tablet_service.cc:1467] Processing CreateTablet for tablet 91a2bf5ddc03431b963ab5da831fd340 (DEFAULT_TABLE table=rebalance_test_table_13 [id=e21b0047cbf842c6833a78db81a34171]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:47.422405 4479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a2bf5ddc03431b963ab5da831fd340. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:47.431056 6373 tablet_bootstrap.cc:492] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:47.431259 6371 tablet_bootstrap.cc:492] T 9d165c29dabe4ae3be11ed84702c3cb2 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:47.433140 6371 tablet_bootstrap.cc:654] T 9d165c29dabe4ae3be11ed84702c3cb2 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:47.433378 6372 tablet_bootstrap.cc:492] T 9d165c29dabe4ae3be11ed84702c3cb2 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:47.435674 6374 tablet_bootstrap.cc:492] T 91a2bf5ddc03431b963ab5da831fd340 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:47.436280 6373 tablet_bootstrap.cc:654] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:47.436903 6372 tablet_bootstrap.cc:654] T 9d165c29dabe4ae3be11ed84702c3cb2 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:47.438967 6374 tablet_bootstrap.cc:654] T 91a2bf5ddc03431b963ab5da831fd340 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:47.442519 6371 tablet_bootstrap.cc:492] T 9d165c29dabe4ae3be11ed84702c3cb2 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:47.442811 6371 ts_tablet_manager.cc:1367] T 9d165c29dabe4ae3be11ed84702c3cb2 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20240501 20:09:47.443939 6371 raft_consensus.cc:348] T 9d165c29dabe4ae3be11ed84702c3cb2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.444375 6371 raft_consensus.cc:374] T 9d165c29dabe4ae3be11ed84702c3cb2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:47.444523 6371 raft_consensus.cc:729] T 9d165c29dabe4ae3be11ed84702c3cb2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:47.444852 6371 consensus_queue.cc:260] T 9d165c29dabe4ae3be11ed84702c3cb2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.445396 6372 tablet_bootstrap.cc:492] T 9d165c29dabe4ae3be11ed84702c3cb2 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:47.445542 6372 ts_tablet_manager.cc:1367] T 9d165c29dabe4ae3be11ed84702c3cb2 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20240501 20:09:47.446321 6372 raft_consensus.cc:348] T 9d165c29dabe4ae3be11ed84702c3cb2 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.446516 6372 raft_consensus.cc:374] T 9d165c29dabe4ae3be11ed84702c3cb2 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:47.446640 6372 raft_consensus.cc:729] T 9d165c29dabe4ae3be11ed84702c3cb2 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:47.446942 6371 ts_tablet_manager.cc:1398] T 9d165c29dabe4ae3be11ed84702c3cb2 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 20:09:47.446874 6372 consensus_queue.cc:260] T 9d165c29dabe4ae3be11ed84702c3cb2 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.447331 6371 tablet_bootstrap.cc:492] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:47.448356 6372 ts_tablet_manager.cc:1398] T 9d165c29dabe4ae3be11ed84702c3cb2 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:09:47.448482 6373 tablet_bootstrap.cc:492] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:47.448626 6372 tablet_bootstrap.cc:492] T 46df0faa681b469abe9e383b181a6053 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:47.448725 6373 ts_tablet_manager.cc:1367] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.004s
I20240501 20:09:47.449251 6371 tablet_bootstrap.cc:654] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:47.450204 6372 tablet_bootstrap.cc:654] T 46df0faa681b469abe9e383b181a6053 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:47.451764 6374 tablet_bootstrap.cc:492] T 91a2bf5ddc03431b963ab5da831fd340 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:47.452395 6373 raft_consensus.cc:348] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.452800 6373 raft_consensus.cc:374] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:47.452970 6373 raft_consensus.cc:729] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:47.453331 6373 consensus_queue.cc:260] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.453840 6374 ts_tablet_manager.cc:1367] T 91a2bf5ddc03431b963ab5da831fd340 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.004s
I20240501 20:09:47.457057 6373 ts_tablet_manager.cc:1398] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240501 20:09:47.457535 6373 tablet_bootstrap.cc:492] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:47.456432 6374 raft_consensus.cc:348] T 91a2bf5ddc03431b963ab5da831fd340 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.459113 6374 raft_consensus.cc:374] T 91a2bf5ddc03431b963ab5da831fd340 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:47.459204 6374 raft_consensus.cc:729] T 91a2bf5ddc03431b963ab5da831fd340 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:47.459461 6374 consensus_queue.cc:260] T 91a2bf5ddc03431b963ab5da831fd340 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.459580 6372 tablet_bootstrap.cc:492] T 46df0faa681b469abe9e383b181a6053 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:47.459741 6372 ts_tablet_manager.cc:1367] T 46df0faa681b469abe9e383b181a6053 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240501 20:09:47.460614 6372 raft_consensus.cc:348] T 46df0faa681b469abe9e383b181a6053 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.460856 6372 raft_consensus.cc:374] T 46df0faa681b469abe9e383b181a6053 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:47.460932 6372 raft_consensus.cc:729] T 46df0faa681b469abe9e383b181a6053 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:47.461246 6372 consensus_queue.cc:260] T 46df0faa681b469abe9e383b181a6053 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.459654 6371 tablet_bootstrap.cc:492] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:47.461747 6371 ts_tablet_manager.cc:1367] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20240501 20:09:47.462193 6374 ts_tablet_manager.cc:1398] T 91a2bf5ddc03431b963ab5da831fd340 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240501 20:09:47.462853 6372 ts_tablet_manager.cc:1398] T 46df0faa681b469abe9e383b181a6053 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 20:09:47.463178 6372 tablet_bootstrap.cc:492] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:47.463552 6371 raft_consensus.cc:348] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.463868 6371 raft_consensus.cc:374] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:47.463941 6371 raft_consensus.cc:729] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:47.464224 6371 consensus_queue.cc:260] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.464819 6371 ts_tablet_manager.cc:1398] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:09:47.465051 6373 tablet_bootstrap.cc:654] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:47.467344 6372 tablet_bootstrap.cc:654] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:47.469435 6373 tablet_bootstrap.cc:492] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:47.469628 6373 ts_tablet_manager.cc:1367] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.001s
I20240501 20:09:47.470734 6372 tablet_bootstrap.cc:492] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:47.470688 6373 raft_consensus.cc:348] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.470889 6372 ts_tablet_manager.cc:1367] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240501 20:09:47.470908 6373 raft_consensus.cc:374] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:47.470986 6373 raft_consensus.cc:729] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:47.471272 6373 consensus_queue.cc:260] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.471840 6373 ts_tablet_manager.cc:1398] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:47.472011 6372 raft_consensus.cc:348] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.472118 6373 tablet_bootstrap.cc:492] T 91a2bf5ddc03431b963ab5da831fd340 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:47.472198 6372 raft_consensus.cc:374] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:47.472299 6372 raft_consensus.cc:729] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:47.472522 6372 consensus_queue.cc:260] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.473115 6372 ts_tablet_manager.cc:1398] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:47.475318 6373 tablet_bootstrap.cc:654] T 91a2bf5ddc03431b963ab5da831fd340 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:47.477636 6373 tablet_bootstrap.cc:492] T 91a2bf5ddc03431b963ab5da831fd340 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:47.477818 6373 ts_tablet_manager.cc:1367] T 91a2bf5ddc03431b963ab5da831fd340 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 20:09:47.478442 6373 raft_consensus.cc:348] T 91a2bf5ddc03431b963ab5da831fd340 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.478590 6373 raft_consensus.cc:374] T 91a2bf5ddc03431b963ab5da831fd340 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:47.478693 6373 raft_consensus.cc:729] T 91a2bf5ddc03431b963ab5da831fd340 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:47.478893 6373 consensus_queue.cc:260] T 91a2bf5ddc03431b963ab5da831fd340 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.479344 6373 ts_tablet_manager.cc:1398] T 91a2bf5ddc03431b963ab5da831fd340 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 20:09:47.502777 6297 raft_consensus.cc:482] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:47.502939 6297 raft_consensus.cc:504] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.503592 6297 leader_election.cc:288] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:09:47.504026 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a2bf5ddc03431b963ab5da831fd340" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:47.504113 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a2bf5ddc03431b963ab5da831fd340" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:47.504271 4234 raft_consensus.cc:2454] T 91a2bf5ddc03431b963ab5da831fd340 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:09:47.504411 4499 raft_consensus.cc:2454] T 91a2bf5ddc03431b963ab5da831fd340 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:09:47.504678 4301 leader_election.cc:302] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:47.504978 6297 raft_consensus.cc:2781] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:47.505095 6297 raft_consensus.cc:482] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:47.505190 6297 raft_consensus.cc:3037] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:47.507153 6297 raft_consensus.cc:504] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.507649 6297 leader_election.cc:288] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Requested vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:09:47.508105 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a2bf5ddc03431b963ab5da831fd340" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:47.508105 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a2bf5ddc03431b963ab5da831fd340" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:47.508304 4234 raft_consensus.cc:3037] T 91a2bf5ddc03431b963ab5da831fd340 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:47.508327 4499 raft_consensus.cc:3037] T 91a2bf5ddc03431b963ab5da831fd340 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:47.510376 4499 raft_consensus.cc:2454] T 91a2bf5ddc03431b963ab5da831fd340 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:09:47.510418 4234 raft_consensus.cc:2454] T 91a2bf5ddc03431b963ab5da831fd340 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:09:47.510851 4301 leader_election.cc:302] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:47.511145 6297 raft_consensus.cc:2781] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:47.511298 6297 raft_consensus.cc:686] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Becoming Leader. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Running, Role: LEADER
I20240501 20:09:47.511579 6297 consensus_queue.cc:237] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.515031 4085 catalog_manager.cc:5484] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a reported cstate change: term changed from 0 to 1, leader changed from <none> to 10b47782558c41ee91f5e164a372049a (127.29.219.66). New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:47.688658 6054 raft_consensus.cc:482] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:47.688880 6054 raft_consensus.cc:504] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.689585 6054 leader_election.cc:288] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:47.690023 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46df0faa681b469abe9e383b181a6053" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:47.690143 4366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46df0faa681b469abe9e383b181a6053" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:47.690263 4630 raft_consensus.cc:2454] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:47.690376 4366 raft_consensus.cc:2454] T 46df0faa681b469abe9e383b181a6053 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:47.690737 4435 leader_election.cc:302] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:47.691015 6054 raft_consensus.cc:2781] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:47.691094 6054 raft_consensus.cc:482] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:47.691176 6054 raft_consensus.cc:3037] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:47.693818 6054 raft_consensus.cc:504] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.694507 6054 leader_election.cc:288] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:47.694998 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46df0faa681b469abe9e383b181a6053" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:47.695062 4366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46df0faa681b469abe9e383b181a6053" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:47.695216 4630 raft_consensus.cc:3037] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:47.695242 4366 raft_consensus.cc:3037] T 46df0faa681b469abe9e383b181a6053 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:47.698099 4366 raft_consensus.cc:2454] T 46df0faa681b469abe9e383b181a6053 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:47.698098 4630 raft_consensus.cc:2454] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:47.698596 4434 leader_election.cc:302] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:47.698951 6054 raft_consensus.cc:2781] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:47.699119 6054 raft_consensus.cc:686] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:09:47.699383 6054 consensus_queue.cc:237] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.702975 4085 catalog_manager.cc:5484] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:47.774674 6054 raft_consensus.cc:482] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:47.774855 6054 raft_consensus.cc:504] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.775885 6054 leader_election.cc:288] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:47.775990 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d165c29dabe4ae3be11ed84702c3cb2" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:47.776005 4366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d165c29dabe4ae3be11ed84702c3cb2" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:47.776225 4630 raft_consensus.cc:2454] T 9d165c29dabe4ae3be11ed84702c3cb2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:47.776238 4366 raft_consensus.cc:2454] T 9d165c29dabe4ae3be11ed84702c3cb2 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:47.776633 4434 leader_election.cc:302] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:47.776907 6054 raft_consensus.cc:2781] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:47.777019 6054 raft_consensus.cc:482] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:47.777099 6054 raft_consensus.cc:3037] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:47.779067 6054 raft_consensus.cc:504] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.779667 6054 leader_election.cc:288] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:47.779984 4366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d165c29dabe4ae3be11ed84702c3cb2" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:47.780177 4366 raft_consensus.cc:3037] T 9d165c29dabe4ae3be11ed84702c3cb2 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:47.780117 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d165c29dabe4ae3be11ed84702c3cb2" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:47.780332 4630 raft_consensus.cc:3037] T 9d165c29dabe4ae3be11ed84702c3cb2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:47.781994 4366 raft_consensus.cc:2454] T 9d165c29dabe4ae3be11ed84702c3cb2 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:47.782222 4630 raft_consensus.cc:2454] T 9d165c29dabe4ae3be11ed84702c3cb2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:47.782380 4434 leader_election.cc:302] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:47.782660 6054 raft_consensus.cc:2781] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:47.782809 6054 raft_consensus.cc:686] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:09:47.783087 6054 consensus_queue.cc:237] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:47.786305 4085 catalog_manager.cc:5484] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:47.998816 6344 consensus_queue.cc:1035] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:48.005632 6297 consensus_queue.cc:1035] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:48.077636 6211 consensus_queue.cc:1035] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:48.082221 6211 consensus_queue.cc:1035] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:48.190817 6054 consensus_queue.cc:1035] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:48.196838 6395 consensus_queue.cc:1035] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:48.247427 6388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:48.247799 6388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:48.255614 6388 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_13'
INSERT report
rows total: 2000
time total: 232.618 ms
time per row: 0.116309 ms
I20240501 20:09:48.651360 4085 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:48.660075 4347 tablet_service.cc:1467] Processing CreateTablet for tablet b0dfb22caa4f4855b4976eb3cb27c681 (DEFAULT_TABLE table=rebalance_test_table_14 [id=62c829b3c0b3400bb175722fa826b0f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:48.660544 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0dfb22caa4f4855b4976eb3cb27c681. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.660583 4609 tablet_service.cc:1467] Processing CreateTablet for tablet b0dfb22caa4f4855b4976eb3cb27c681 (DEFAULT_TABLE table=rebalance_test_table_14 [id=62c829b3c0b3400bb175722fa826b0f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:48.660959 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0dfb22caa4f4855b4976eb3cb27c681. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.661160 4611 tablet_service.cc:1467] Processing CreateTablet for tablet a215d2ce50c347efb5b536a27d037094 (DEFAULT_TABLE table=rebalance_test_table_14 [id=62c829b3c0b3400bb175722fa826b0f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:48.661479 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a215d2ce50c347efb5b536a27d037094. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.666986 4213 tablet_service.cc:1467] Processing CreateTablet for tablet b0dfb22caa4f4855b4976eb3cb27c681 (DEFAULT_TABLE table=rebalance_test_table_14 [id=62c829b3c0b3400bb175722fa826b0f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:48.667325 6454 tablet_bootstrap.cc:492] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:48.667429 4213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0dfb22caa4f4855b4976eb3cb27c681. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.667858 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 8add007651394975a012f04fbf7241ed (DEFAULT_TABLE table=rebalance_test_table_14 [id=62c829b3c0b3400bb175722fa826b0f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:48.668239 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8add007651394975a012f04fbf7241ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.669174 6454 tablet_bootstrap.cc:654] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:48.671298 4479 tablet_service.cc:1467] Processing CreateTablet for tablet a215d2ce50c347efb5b536a27d037094 (DEFAULT_TABLE table=rebalance_test_table_14 [id=62c829b3c0b3400bb175722fa826b0f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:48.671742 4479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a215d2ce50c347efb5b536a27d037094. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.672271 4348 tablet_service.cc:1467] Processing CreateTablet for tablet 8add007651394975a012f04fbf7241ed (DEFAULT_TABLE table=rebalance_test_table_14 [id=62c829b3c0b3400bb175722fa826b0f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:48.672734 4348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8add007651394975a012f04fbf7241ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.675128 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 8add007651394975a012f04fbf7241ed (DEFAULT_TABLE table=rebalance_test_table_14 [id=62c829b3c0b3400bb175722fa826b0f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:48.675663 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8add007651394975a012f04fbf7241ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.680138 4214 tablet_service.cc:1467] Processing CreateTablet for tablet a215d2ce50c347efb5b536a27d037094 (DEFAULT_TABLE table=rebalance_test_table_14 [id=62c829b3c0b3400bb175722fa826b0f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:48.680665 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a215d2ce50c347efb5b536a27d037094. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:48.700450 6454 tablet_bootstrap.cc:492] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:48.700626 6454 ts_tablet_manager.cc:1367] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.010s
I20240501 20:09:48.701558 6454 raft_consensus.cc:348] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:48.701838 6454 raft_consensus.cc:374] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:48.701936 6454 raft_consensus.cc:729] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:48.702221 6454 consensus_queue.cc:260] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:48.724198 6455 tablet_bootstrap.cc:492] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:48.724788 6457 tablet_bootstrap.cc:492] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:48.726537 6457 tablet_bootstrap.cc:654] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:48.728282 6454 ts_tablet_manager.cc:1398] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
I20240501 20:09:48.728736 6455 tablet_bootstrap.cc:654] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:48.729197 6454 tablet_bootstrap.cc:492] T 8add007651394975a012f04fbf7241ed P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:48.730441 6459 tablet_bootstrap.cc:492] T a215d2ce50c347efb5b536a27d037094 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:48.732105 6459 tablet_bootstrap.cc:654] T a215d2ce50c347efb5b536a27d037094 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:48.734318 6459 tablet_bootstrap.cc:492] T a215d2ce50c347efb5b536a27d037094 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:48.734549 6459 ts_tablet_manager.cc:1367] T a215d2ce50c347efb5b536a27d037094 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 20:09:48.735641 6459 raft_consensus.cc:348] T a215d2ce50c347efb5b536a27d037094 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:48.735992 6459 raft_consensus.cc:374] T a215d2ce50c347efb5b536a27d037094 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:48.736152 6459 raft_consensus.cc:729] T a215d2ce50c347efb5b536a27d037094 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:48.736461 6459 consensus_queue.cc:260] T a215d2ce50c347efb5b536a27d037094 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:48.737926 6454 tablet_bootstrap.cc:654] T 8add007651394975a012f04fbf7241ed P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:48.743409 6459 ts_tablet_manager.cc:1398] T a215d2ce50c347efb5b536a27d037094 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240501 20:09:48.746956 6457 tablet_bootstrap.cc:492] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:48.747238 6457 ts_tablet_manager.cc:1367] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.007s
I20240501 20:09:48.748198 6457 raft_consensus.cc:348] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:48.748405 6457 raft_consensus.cc:374] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:48.748494 6457 raft_consensus.cc:729] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:48.748736 6457 consensus_queue.cc:260] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:48.749464 6457 ts_tablet_manager.cc:1398] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:09:48.749858 6457 tablet_bootstrap.cc:492] T b0dfb22caa4f4855b4976eb3cb27c681 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:48.751835 6457 tablet_bootstrap.cc:654] T b0dfb22caa4f4855b4976eb3cb27c681 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:48.754765 6454 tablet_bootstrap.cc:492] T 8add007651394975a012f04fbf7241ed P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:48.755415 6454 ts_tablet_manager.cc:1367] T 8add007651394975a012f04fbf7241ed P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.001s
I20240501 20:09:48.756420 6457 tablet_bootstrap.cc:492] T b0dfb22caa4f4855b4976eb3cb27c681 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:48.756323 6454 raft_consensus.cc:348] T 8add007651394975a012f04fbf7241ed P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:48.756645 6457 ts_tablet_manager.cc:1367] T b0dfb22caa4f4855b4976eb3cb27c681 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240501 20:09:48.756652 6454 raft_consensus.cc:374] T 8add007651394975a012f04fbf7241ed P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:48.756727 6454 raft_consensus.cc:729] T 8add007651394975a012f04fbf7241ed P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:48.756969 6454 consensus_queue.cc:260] T 8add007651394975a012f04fbf7241ed P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:48.757597 6454 ts_tablet_manager.cc:1398] T 8add007651394975a012f04fbf7241ed P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:48.757555 6457 raft_consensus.cc:348] T b0dfb22caa4f4855b4976eb3cb27c681 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:48.757755 6457 raft_consensus.cc:374] T b0dfb22caa4f4855b4976eb3cb27c681 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:48.757822 6457 raft_consensus.cc:729] T b0dfb22caa4f4855b4976eb3cb27c681 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:48.758554 6455 tablet_bootstrap.cc:492] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:48.758726 6455 ts_tablet_manager.cc:1367] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.008s
I20240501 20:09:48.759487 6455 raft_consensus.cc:348] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:48.759713 6455 raft_consensus.cc:374] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:48.759819 6455 raft_consensus.cc:729] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:48.760038 6455 consensus_queue.cc:260] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:48.760596 6455 ts_tablet_manager.cc:1398] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:48.760869 6455 tablet_bootstrap.cc:492] T b0dfb22caa4f4855b4976eb3cb27c681 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:48.763072 6457 consensus_queue.cc:260] T b0dfb22caa4f4855b4976eb3cb27c681 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:48.764714 6455 tablet_bootstrap.cc:654] T b0dfb22caa4f4855b4976eb3cb27c681 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:48.766927 6457 ts_tablet_manager.cc:1398] T b0dfb22caa4f4855b4976eb3cb27c681 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240501 20:09:48.770524 6457 tablet_bootstrap.cc:492] T a215d2ce50c347efb5b536a27d037094 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:48.774441 6457 tablet_bootstrap.cc:654] T a215d2ce50c347efb5b536a27d037094 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:48.776775 6457 tablet_bootstrap.cc:492] T a215d2ce50c347efb5b536a27d037094 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:48.776930 6457 ts_tablet_manager.cc:1367] T a215d2ce50c347efb5b536a27d037094 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240501 20:09:48.777097 6455 tablet_bootstrap.cc:492] T b0dfb22caa4f4855b4976eb3cb27c681 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:48.777274 6455 ts_tablet_manager.cc:1367] T b0dfb22caa4f4855b4976eb3cb27c681 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.008s
I20240501 20:09:48.777639 6457 raft_consensus.cc:348] T a215d2ce50c347efb5b536a27d037094 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:48.777807 6457 raft_consensus.cc:374] T a215d2ce50c347efb5b536a27d037094 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:48.777928 6457 raft_consensus.cc:729] T a215d2ce50c347efb5b536a27d037094 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:48.778038 6455 raft_consensus.cc:348] T b0dfb22caa4f4855b4976eb3cb27c681 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:48.778241 6455 raft_consensus.cc:374] T b0dfb22caa4f4855b4976eb3cb27c681 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:48.778154 6457 consensus_queue.cc:260] T a215d2ce50c347efb5b536a27d037094 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:48.778312 6455 raft_consensus.cc:729] T b0dfb22caa4f4855b4976eb3cb27c681 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:48.778544 6455 consensus_queue.cc:260] T b0dfb22caa4f4855b4976eb3cb27c681 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:48.778735 6457 ts_tablet_manager.cc:1398] T a215d2ce50c347efb5b536a27d037094 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 20:09:48.779198 6455 ts_tablet_manager.cc:1398] T b0dfb22caa4f4855b4976eb3cb27c681 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:09:48.779645 6455 tablet_bootstrap.cc:492] T 8add007651394975a012f04fbf7241ed P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:48.782485 6455 tablet_bootstrap.cc:654] T 8add007651394975a012f04fbf7241ed P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:48.786844 6455 tablet_bootstrap.cc:492] T 8add007651394975a012f04fbf7241ed P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:48.786969 6455 ts_tablet_manager.cc:1367] T 8add007651394975a012f04fbf7241ed P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 20:09:48.787595 6455 raft_consensus.cc:348] T 8add007651394975a012f04fbf7241ed P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:48.787744 6455 raft_consensus.cc:374] T 8add007651394975a012f04fbf7241ed P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:48.787825 6455 raft_consensus.cc:729] T 8add007651394975a012f04fbf7241ed P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:48.788020 6455 consensus_queue.cc:260] T 8add007651394975a012f04fbf7241ed P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:48.788492 6455 ts_tablet_manager.cc:1398] T 8add007651394975a012f04fbf7241ed P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:09:48.847390 6297 raft_consensus.cc:482] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:48.847579 6297 raft_consensus.cc:504] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:48.848214 6297 leader_election.cc:288] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:48.848670 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0dfb22caa4f4855b4976eb3cb27c681" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:48.848773 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0dfb22caa4f4855b4976eb3cb27c681" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:48.848896 4235 raft_consensus.cc:2454] T b0dfb22caa4f4855b4976eb3cb27c681 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:09:48.849010 4631 raft_consensus.cc:2454] T b0dfb22caa4f4855b4976eb3cb27c681 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:09:48.849372 4301 leader_election.cc:302] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:48.849711 6297 raft_consensus.cc:2781] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:48.849833 6297 raft_consensus.cc:482] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:48.849925 6297 raft_consensus.cc:3037] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:48.851962 6297 raft_consensus.cc:504] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:48.852499 6297 leader_election.cc:288] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Requested vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:48.852926 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0dfb22caa4f4855b4976eb3cb27c681" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:48.852944 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0dfb22caa4f4855b4976eb3cb27c681" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:48.853125 4631 raft_consensus.cc:3037] T b0dfb22caa4f4855b4976eb3cb27c681 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:48.853149 4235 raft_consensus.cc:3037] T b0dfb22caa4f4855b4976eb3cb27c681 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:48.855930 4631 raft_consensus.cc:2454] T b0dfb22caa4f4855b4976eb3cb27c681 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:09:48.855918 4235 raft_consensus.cc:2454] T b0dfb22caa4f4855b4976eb3cb27c681 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:09:48.856319 4301 leader_election.cc:302] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:48.856592 6297 raft_consensus.cc:2781] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:48.856734 6297 raft_consensus.cc:686] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Becoming Leader. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Running, Role: LEADER
I20240501 20:09:48.857018 6297 consensus_queue.cc:237] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:48.860245 4084 catalog_manager.cc:5484] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a reported cstate change: term changed from 0 to 1, leader changed from <none> to 10b47782558c41ee91f5e164a372049a (127.29.219.66). New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:48.889732 6269 raft_consensus.cc:482] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:48.889922 6269 raft_consensus.cc:504] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:48.890558 6269 leader_election.cc:288] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:09:48.891129 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a215d2ce50c347efb5b536a27d037094" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:48.891223 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a215d2ce50c347efb5b536a27d037094" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:48.891322 4499 raft_consensus.cc:2454] T a215d2ce50c347efb5b536a27d037094 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:09:48.891451 4235 raft_consensus.cc:2454] T a215d2ce50c347efb5b536a27d037094 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:09:48.891795 4564 leader_election.cc:302] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:48.892073 6269 raft_consensus.cc:2781] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:48.892179 6269 raft_consensus.cc:482] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:48.892273 6269 raft_consensus.cc:3037] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:48.894588 6269 raft_consensus.cc:504] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:48.895128 6269 leader_election.cc:288] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:09:48.895540 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a215d2ce50c347efb5b536a27d037094" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:48.895614 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a215d2ce50c347efb5b536a27d037094" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:48.895706 4499 raft_consensus.cc:3037] T a215d2ce50c347efb5b536a27d037094 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:48.895815 4235 raft_consensus.cc:3037] T a215d2ce50c347efb5b536a27d037094 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:48.897570 4499 raft_consensus.cc:2454] T a215d2ce50c347efb5b536a27d037094 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:09:48.897657 4235 raft_consensus.cc:2454] T a215d2ce50c347efb5b536a27d037094 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:09:48.897953 4564 leader_election.cc:302] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:48.898228 6461 raft_consensus.cc:2781] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:48.898391 6461 raft_consensus.cc:686] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Becoming Leader. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Running, Role: LEADER
I20240501 20:09:48.898783 6461 consensus_queue.cc:237] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:48.902338 4084 catalog_manager.cc:5484] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68). New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:48.943405 6450 raft_consensus.cc:482] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:48.943583 6450 raft_consensus.cc:504] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:48.944365 6450 leader_election.cc:288] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:48.944721 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8add007651394975a012f04fbf7241ed" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:48.944895 4366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8add007651394975a012f04fbf7241ed" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:48.944973 4631 raft_consensus.cc:2454] T 8add007651394975a012f04fbf7241ed P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:48.945128 4366 raft_consensus.cc:2454] T 8add007651394975a012f04fbf7241ed P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:48.945370 4170 leader_election.cc:302] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:48.945641 6450 raft_consensus.cc:2781] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:48.945777 6450 raft_consensus.cc:482] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:48.945860 6450 raft_consensus.cc:3037] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:48.947804 6450 raft_consensus.cc:504] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:48.948321 6450 leader_election.cc:288] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:48.948786 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8add007651394975a012f04fbf7241ed" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:48.948802 4366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8add007651394975a012f04fbf7241ed" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:48.948987 4366 raft_consensus.cc:3037] T 8add007651394975a012f04fbf7241ed P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:48.948988 4631 raft_consensus.cc:3037] T 8add007651394975a012f04fbf7241ed P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:48.951018 4366 raft_consensus.cc:2454] T 8add007651394975a012f04fbf7241ed P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:48.951021 4631 raft_consensus.cc:2454] T 8add007651394975a012f04fbf7241ed P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:48.951457 4169 leader_election.cc:302] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:48.951800 6450 raft_consensus.cc:2781] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:48.951977 6450 raft_consensus.cc:686] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:09:48.952472 6450 consensus_queue.cc:237] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:48.955687 4084 catalog_manager.cc:5484] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:49.275888 6297 consensus_queue.cc:1035] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:49.288645 6297 consensus_queue.cc:1035] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:49.325765 6269 consensus_queue.cc:1035] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:49.331843 6269 consensus_queue.cc:1035] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:49.372184 6472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:49.372524 6472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:49.380275 6472 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_14'
I20240501 20:09:49.463196 6450 consensus_queue.cc:1035] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:49.473469 6458 consensus_queue.cc:1035] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 319.939 ms
time per row: 0.15997 ms
I20240501 20:09:49.851091 4084 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:49.859589 4347 tablet_service.cc:1467] Processing CreateTablet for tablet 56cb0964dabd41079d4428787efc9a9d (DEFAULT_TABLE table=rebalance_test_table_15 [id=2478c1c300d14f2a9826930dc16a5dec]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:49.859707 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 56cb0964dabd41079d4428787efc9a9d (DEFAULT_TABLE table=rebalance_test_table_15 [id=2478c1c300d14f2a9826930dc16a5dec]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:49.860149 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56cb0964dabd41079d4428787efc9a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:49.860146 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56cb0964dabd41079d4428787efc9a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:49.863580 4611 tablet_service.cc:1467] Processing CreateTablet for tablet dfaa3794205d482c90aa4f860fa99cf3 (DEFAULT_TABLE table=rebalance_test_table_15 [id=2478c1c300d14f2a9826930dc16a5dec]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:49.863620 4479 tablet_service.cc:1467] Processing CreateTablet for tablet 56cb0964dabd41079d4428787efc9a9d (DEFAULT_TABLE table=rebalance_test_table_15 [id=2478c1c300d14f2a9826930dc16a5dec]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:49.863910 4348 tablet_service.cc:1467] Processing CreateTablet for tablet c31148e9e4dd42c3a11787d9384c79c2 (DEFAULT_TABLE table=rebalance_test_table_15 [id=2478c1c300d14f2a9826930dc16a5dec]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:49.864022 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfaa3794205d482c90aa4f860fa99cf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:49.864032 4479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56cb0964dabd41079d4428787efc9a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:49.864271 4348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31148e9e4dd42c3a11787d9384c79c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:49.867007 4478 tablet_service.cc:1467] Processing CreateTablet for tablet dfaa3794205d482c90aa4f860fa99cf3 (DEFAULT_TABLE table=rebalance_test_table_15 [id=2478c1c300d14f2a9826930dc16a5dec]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:49.867432 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfaa3794205d482c90aa4f860fa99cf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:49.871189 4477 tablet_service.cc:1467] Processing CreateTablet for tablet c31148e9e4dd42c3a11787d9384c79c2 (DEFAULT_TABLE table=rebalance_test_table_15 [id=2478c1c300d14f2a9826930dc16a5dec]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:49.871610 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31148e9e4dd42c3a11787d9384c79c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:49.874910 6539 tablet_bootstrap.cc:492] T dfaa3794205d482c90aa4f860fa99cf3 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:49.875095 4610 tablet_service.cc:1467] Processing CreateTablet for tablet c31148e9e4dd42c3a11787d9384c79c2 (DEFAULT_TABLE table=rebalance_test_table_15 [id=2478c1c300d14f2a9826930dc16a5dec]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:49.875506 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31148e9e4dd42c3a11787d9384c79c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:49.877045 6539 tablet_bootstrap.cc:654] T dfaa3794205d482c90aa4f860fa99cf3 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:49.879263 4214 tablet_service.cc:1467] Processing CreateTablet for tablet dfaa3794205d482c90aa4f860fa99cf3 (DEFAULT_TABLE table=rebalance_test_table_15 [id=2478c1c300d14f2a9826930dc16a5dec]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:49.879846 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfaa3794205d482c90aa4f860fa99cf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:49.882988 6539 tablet_bootstrap.cc:492] T dfaa3794205d482c90aa4f860fa99cf3 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:49.883150 6539 ts_tablet_manager.cc:1367] T dfaa3794205d482c90aa4f860fa99cf3 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 20:09:49.884828 6539 raft_consensus.cc:348] T dfaa3794205d482c90aa4f860fa99cf3 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:49.885998 6539 raft_consensus.cc:374] T dfaa3794205d482c90aa4f860fa99cf3 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:49.886128 6539 raft_consensus.cc:729] T dfaa3794205d482c90aa4f860fa99cf3 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:49.886294 6540 tablet_bootstrap.cc:492] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:49.886390 6539 consensus_queue.cc:260] T dfaa3794205d482c90aa4f860fa99cf3 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:49.887086 6539 ts_tablet_manager.cc:1398] T dfaa3794205d482c90aa4f860fa99cf3 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:49.890004 6540 tablet_bootstrap.cc:654] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:49.908376 6539 tablet_bootstrap.cc:492] T c31148e9e4dd42c3a11787d9384c79c2 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:49.910354 6539 tablet_bootstrap.cc:654] T c31148e9e4dd42c3a11787d9384c79c2 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:49.911737 6543 tablet_bootstrap.cc:492] T 56cb0964dabd41079d4428787efc9a9d P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:49.913966 6543 tablet_bootstrap.cc:654] T 56cb0964dabd41079d4428787efc9a9d P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:49.915971 6538 tablet_bootstrap.cc:492] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:49.917304 6540 tablet_bootstrap.cc:492] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:49.917459 6540 ts_tablet_manager.cc:1367] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.031s user 0.014s sys 0.004s
I20240501 20:09:49.918102 6540 raft_consensus.cc:348] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:49.918272 6540 raft_consensus.cc:374] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:49.918390 6540 raft_consensus.cc:729] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:49.918668 6540 consensus_queue.cc:260] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:49.919411 6538 tablet_bootstrap.cc:654] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:49.919592 6540 ts_tablet_manager.cc:1398] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:49.920037 6540 tablet_bootstrap.cc:492] T 56cb0964dabd41079d4428787efc9a9d P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:49.925931 6540 tablet_bootstrap.cc:654] T 56cb0964dabd41079d4428787efc9a9d P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:49.931979 6543 tablet_bootstrap.cc:492] T 56cb0964dabd41079d4428787efc9a9d P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:49.932024 6540 tablet_bootstrap.cc:492] T 56cb0964dabd41079d4428787efc9a9d P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:49.932204 6540 ts_tablet_manager.cc:1367] T 56cb0964dabd41079d4428787efc9a9d P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20240501 20:09:49.932219 6538 tablet_bootstrap.cc:492] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:49.932258 6543 ts_tablet_manager.cc:1367] T 56cb0964dabd41079d4428787efc9a9d P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.009s
I20240501 20:09:49.932356 6538 ts_tablet_manager.cc:1367] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20240501 20:09:49.932904 6539 tablet_bootstrap.cc:492] T c31148e9e4dd42c3a11787d9384c79c2 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:49.933166 6539 ts_tablet_manager.cc:1367] T c31148e9e4dd42c3a11787d9384c79c2 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.006s
I20240501 20:09:49.933117 6540 raft_consensus.cc:348] T 56cb0964dabd41079d4428787efc9a9d P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:49.933246 6543 raft_consensus.cc:348] T 56cb0964dabd41079d4428787efc9a9d P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:49.933422 6540 raft_consensus.cc:374] T 56cb0964dabd41079d4428787efc9a9d P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:49.933528 6540 raft_consensus.cc:729] T 56cb0964dabd41079d4428787efc9a9d P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:49.933562 6543 raft_consensus.cc:374] T 56cb0964dabd41079d4428787efc9a9d P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:49.933719 6543 raft_consensus.cc:729] T 56cb0964dabd41079d4428787efc9a9d P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:49.933830 6540 consensus_queue.cc:260] T 56cb0964dabd41079d4428787efc9a9d P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:49.933815 6538 raft_consensus.cc:348] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:49.934116 6538 raft_consensus.cc:374] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:49.934051 6543 consensus_queue.cc:260] T 56cb0964dabd41079d4428787efc9a9d P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:49.934252 6538 raft_consensus.cc:729] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:49.934559 6540 ts_tablet_manager.cc:1398] T 56cb0964dabd41079d4428787efc9a9d P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:09:49.934648 6538 consensus_queue.cc:260] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:49.934876 6543 ts_tablet_manager.cc:1398] T 56cb0964dabd41079d4428787efc9a9d P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:49.935180 6543 tablet_bootstrap.cc:492] T c31148e9e4dd42c3a11787d9384c79c2 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:49.936322 6539 raft_consensus.cc:348] T c31148e9e4dd42c3a11787d9384c79c2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:49.936558 6539 raft_consensus.cc:374] T c31148e9e4dd42c3a11787d9384c79c2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:49.936630 6539 raft_consensus.cc:729] T c31148e9e4dd42c3a11787d9384c79c2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:49.936854 6539 consensus_queue.cc:260] T c31148e9e4dd42c3a11787d9384c79c2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:49.937477 6539 ts_tablet_manager.cc:1398] T c31148e9e4dd42c3a11787d9384c79c2 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 20:09:49.940357 6538 ts_tablet_manager.cc:1398] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 20:09:49.940771 6538 tablet_bootstrap.cc:492] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:49.942262 6543 tablet_bootstrap.cc:654] T c31148e9e4dd42c3a11787d9384c79c2 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:49.942725 6538 tablet_bootstrap.cc:654] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:49.945609 6538 tablet_bootstrap.cc:492] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:49.945766 6538 ts_tablet_manager.cc:1367] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240501 20:09:49.946311 6543 tablet_bootstrap.cc:492] T c31148e9e4dd42c3a11787d9384c79c2 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:49.946477 6543 ts_tablet_manager.cc:1367] T c31148e9e4dd42c3a11787d9384c79c2 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240501 20:09:49.946700 6538 raft_consensus.cc:348] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:49.946929 6538 raft_consensus.cc:374] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:49.947013 6538 raft_consensus.cc:729] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:49.947299 6538 consensus_queue.cc:260] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:49.947325 6543 raft_consensus.cc:348] T c31148e9e4dd42c3a11787d9384c79c2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:49.947515 6543 raft_consensus.cc:374] T c31148e9e4dd42c3a11787d9384c79c2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:49.947875 6538 ts_tablet_manager.cc:1398] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:49.947595 6543 raft_consensus.cc:729] T c31148e9e4dd42c3a11787d9384c79c2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:49.948701 6543 consensus_queue.cc:260] T c31148e9e4dd42c3a11787d9384c79c2 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:49.949458 6543 ts_tablet_manager.cc:1398] T c31148e9e4dd42c3a11787d9384c79c2 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 20:09:49.949780 6543 tablet_bootstrap.cc:492] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:49.951444 6543 tablet_bootstrap.cc:654] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:49.954133 6543 tablet_bootstrap.cc:492] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:49.954272 6543 ts_tablet_manager.cc:1367] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:09:49.955677 6543 raft_consensus.cc:348] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:49.955844 6543 raft_consensus.cc:374] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:49.955938 6543 raft_consensus.cc:729] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:49.956127 6543 consensus_queue.cc:260] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:49.956614 6543 ts_tablet_manager.cc:1398] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:09:50.030957 6442 raft_consensus.cc:482] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:50.031140 6442 raft_consensus.cc:504] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:50.031826 6442 leader_election.cc:288] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:50.032333 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31148e9e4dd42c3a11787d9384c79c2" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:50.032407 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31148e9e4dd42c3a11787d9384c79c2" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:50.032599 4499 raft_consensus.cc:2454] T c31148e9e4dd42c3a11787d9384c79c2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:09:50.032627 4630 raft_consensus.cc:2454] T c31148e9e4dd42c3a11787d9384c79c2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:09:50.033073 4301 leader_election.cc:302] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:50.033361 6442 raft_consensus.cc:2781] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:50.033484 6442 raft_consensus.cc:482] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:50.033577 6442 raft_consensus.cc:3037] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:50.035652 6442 raft_consensus.cc:504] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:50.036253 6442 leader_election.cc:288] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:50.036769 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31148e9e4dd42c3a11787d9384c79c2" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:50.036773 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31148e9e4dd42c3a11787d9384c79c2" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:50.036975 4630 raft_consensus.cc:3037] T c31148e9e4dd42c3a11787d9384c79c2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:50.037003 4499 raft_consensus.cc:3037] T c31148e9e4dd42c3a11787d9384c79c2 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:50.039856 4499 raft_consensus.cc:2454] T c31148e9e4dd42c3a11787d9384c79c2 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:09:50.039891 4630 raft_consensus.cc:2454] T c31148e9e4dd42c3a11787d9384c79c2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:09:50.040299 4301 leader_election.cc:302] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:50.040637 6442 raft_consensus.cc:2781] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:50.040827 6442 raft_consensus.cc:686] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Becoming Leader. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Running, Role: LEADER
I20240501 20:09:50.041144 6442 consensus_queue.cc:237] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:50.044160 4087 catalog_manager.cc:5484] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a reported cstate change: term changed from 0 to 1, leader changed from <none> to 10b47782558c41ee91f5e164a372049a (127.29.219.66). New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:50.062327 6520 raft_consensus.cc:482] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:50.062538 6520 raft_consensus.cc:504] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:50.063402 6520 leader_election.cc:288] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:09:50.063804 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfaa3794205d482c90aa4f860fa99cf3" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:50.063902 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfaa3794205d482c90aa4f860fa99cf3" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:50.064011 4630 raft_consensus.cc:2454] T dfaa3794205d482c90aa4f860fa99cf3 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:50.064136 4499 raft_consensus.cc:2454] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:50.064374 4170 leader_election.cc:302] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:50.064630 6520 raft_consensus.cc:2781] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:50.064750 6520 raft_consensus.cc:482] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:50.064823 6520 raft_consensus.cc:3037] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:50.066746 6520 raft_consensus.cc:504] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:50.067277 6520 leader_election.cc:288] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:09:50.067713 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfaa3794205d482c90aa4f860fa99cf3" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:50.067741 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfaa3794205d482c90aa4f860fa99cf3" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:50.067891 4499 raft_consensus.cc:3037] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:50.067919 4630 raft_consensus.cc:3037] T dfaa3794205d482c90aa4f860fa99cf3 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:50.069720 4630 raft_consensus.cc:2454] T dfaa3794205d482c90aa4f860fa99cf3 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:50.069815 4499 raft_consensus.cc:2454] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:50.070195 4170 leader_election.cc:302] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:50.070490 6520 raft_consensus.cc:2781] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:50.070669 6520 raft_consensus.cc:686] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:09:50.070951 6520 consensus_queue.cc:237] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:50.074111 4087 catalog_manager.cc:5484] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:50.091297 6520 raft_consensus.cc:482] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:50.091516 6520 raft_consensus.cc:504] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:50.092211 6520 leader_election.cc:288] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:09:50.092621 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cb0964dabd41079d4428787efc9a9d" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:50.092883 4365 raft_consensus.cc:2454] T 56cb0964dabd41079d4428787efc9a9d P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:50.093077 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cb0964dabd41079d4428787efc9a9d" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:50.093310 4499 raft_consensus.cc:2454] T 56cb0964dabd41079d4428787efc9a9d P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:50.093351 4169 leader_election.cc:302] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:50.093653 6520 raft_consensus.cc:2781] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:50.093761 6520 raft_consensus.cc:482] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:50.093873 6520 raft_consensus.cc:3037] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:50.095927 6520 raft_consensus.cc:504] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:50.096494 6520 leader_election.cc:288] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:09:50.096863 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cb0964dabd41079d4428787efc9a9d" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:50.096961 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56cb0964dabd41079d4428787efc9a9d" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:50.097087 4365 raft_consensus.cc:3037] T 56cb0964dabd41079d4428787efc9a9d P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:50.097206 4499 raft_consensus.cc:3037] T 56cb0964dabd41079d4428787efc9a9d P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:50.100158 4365 raft_consensus.cc:2454] T 56cb0964dabd41079d4428787efc9a9d P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:50.100157 4499 raft_consensus.cc:2454] T 56cb0964dabd41079d4428787efc9a9d P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:50.100611 4169 leader_election.cc:302] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:50.100904 6520 raft_consensus.cc:2781] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:50.101078 6520 raft_consensus.cc:686] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:09:50.101330 6520 consensus_queue.cc:237] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:50.104554 4087 catalog_manager.cc:5484] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:50.448356 6442 consensus_queue.cc:1035] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:50.457675 6442 consensus_queue.cc:1035] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:50.526381 6554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:50.526779 6554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:50.536058 6554 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:09:50.537061 6520 consensus_queue.cc:1035] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:50.551092 6520 consensus_queue.cc:1035] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:50.561937 6458 consensus_queue.cc:1035] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:09:50.574604 6458 consensus_queue.cc:1035] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_15'
W20240501 20:09:50.702778 4564 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.67:34595, user_credentials={real_user=slave}} blocked reactor thread for 56880.7us
INSERT report
rows total: 2000
time total: 319.54 ms
time per row: 0.15977 ms
I20240501 20:09:51.034056 4083 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:51.044179 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 66c533992f8b4c7a937e79f397636bcc (DEFAULT_TABLE table=rebalance_test_table_16 [id=e07c82123bdc4376bae10c48e6e6543a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:51.044709 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66c533992f8b4c7a937e79f397636bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:51.044709 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 5d9602e8417d444b828c6401632bffed (DEFAULT_TABLE table=rebalance_test_table_16 [id=e07c82123bdc4376bae10c48e6e6543a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:51.044816 4347 tablet_service.cc:1467] Processing CreateTablet for tablet 66c533992f8b4c7a937e79f397636bcc (DEFAULT_TABLE table=rebalance_test_table_16 [id=e07c82123bdc4376bae10c48e6e6543a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:51.044907 4348 tablet_service.cc:1467] Processing CreateTablet for tablet 5d9602e8417d444b828c6401632bffed (DEFAULT_TABLE table=rebalance_test_table_16 [id=e07c82123bdc4376bae10c48e6e6543a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:51.045128 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d9602e8417d444b828c6401632bffed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:51.045218 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66c533992f8b4c7a937e79f397636bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:51.045495 4348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d9602e8417d444b828c6401632bffed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:51.046602 4214 tablet_service.cc:1467] Processing CreateTablet for tablet 5d9602e8417d444b828c6401632bffed (DEFAULT_TABLE table=rebalance_test_table_16 [id=e07c82123bdc4376bae10c48e6e6543a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:51.047065 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d9602e8417d444b828c6401632bffed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:51.051105 4479 tablet_service.cc:1467] Processing CreateTablet for tablet 66c533992f8b4c7a937e79f397636bcc (DEFAULT_TABLE table=rebalance_test_table_16 [id=e07c82123bdc4376bae10c48e6e6543a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:51.051558 4479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66c533992f8b4c7a937e79f397636bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:51.065776 4478 tablet_service.cc:1467] Processing CreateTablet for tablet db31c514cd704786b4be2217daaa11d5 (DEFAULT_TABLE table=rebalance_test_table_16 [id=e07c82123bdc4376bae10c48e6e6543a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:51.065950 4611 tablet_service.cc:1467] Processing CreateTablet for tablet db31c514cd704786b4be2217daaa11d5 (DEFAULT_TABLE table=rebalance_test_table_16 [id=e07c82123bdc4376bae10c48e6e6543a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:51.066519 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db31c514cd704786b4be2217daaa11d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:51.067514 4213 tablet_service.cc:1467] Processing CreateTablet for tablet db31c514cd704786b4be2217daaa11d5 (DEFAULT_TABLE table=rebalance_test_table_16 [id=e07c82123bdc4376bae10c48e6e6543a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:51.067765 6621 tablet_bootstrap.cc:492] T 66c533992f8b4c7a937e79f397636bcc P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:51.068037 4213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db31c514cd704786b4be2217daaa11d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:51.069747 6621 tablet_bootstrap.cc:654] T 66c533992f8b4c7a937e79f397636bcc P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:51.076962 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db31c514cd704786b4be2217daaa11d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:51.080685 6621 tablet_bootstrap.cc:492] T 66c533992f8b4c7a937e79f397636bcc P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:51.080859 6621 ts_tablet_manager.cc:1367] T 66c533992f8b4c7a937e79f397636bcc P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.010s
I20240501 20:09:51.081718 6621 raft_consensus.cc:348] T 66c533992f8b4c7a937e79f397636bcc P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:51.081923 6621 raft_consensus.cc:374] T 66c533992f8b4c7a937e79f397636bcc P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:51.082000 6621 raft_consensus.cc:729] T 66c533992f8b4c7a937e79f397636bcc P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:51.082245 6621 consensus_queue.cc:260] T 66c533992f8b4c7a937e79f397636bcc P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:51.086570 6621 ts_tablet_manager.cc:1398] T 66c533992f8b4c7a937e79f397636bcc P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 20:09:51.091699 6621 tablet_bootstrap.cc:492] T 5d9602e8417d444b828c6401632bffed P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:51.093607 6621 tablet_bootstrap.cc:654] T 5d9602e8417d444b828c6401632bffed P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:51.095455 6623 tablet_bootstrap.cc:492] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:51.098945 6623 tablet_bootstrap.cc:654] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:51.102149 6619 tablet_bootstrap.cc:492] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:51.102907 6620 tablet_bootstrap.cc:492] T 5d9602e8417d444b828c6401632bffed P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:51.105895 6621 tablet_bootstrap.cc:492] T 5d9602e8417d444b828c6401632bffed P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:51.106087 6621 ts_tablet_manager.cc:1367] T 5d9602e8417d444b828c6401632bffed P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20240501 20:09:51.106945 6620 tablet_bootstrap.cc:654] T 5d9602e8417d444b828c6401632bffed P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:51.106914 6621 raft_consensus.cc:348] T 5d9602e8417d444b828c6401632bffed P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:51.107178 6621 raft_consensus.cc:374] T 5d9602e8417d444b828c6401632bffed P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:51.107260 6621 raft_consensus.cc:729] T 5d9602e8417d444b828c6401632bffed P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:51.107525 6621 consensus_queue.cc:260] T 5d9602e8417d444b828c6401632bffed P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:51.107945 6619 tablet_bootstrap.cc:654] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:51.108275 6621 ts_tablet_manager.cc:1398] T 5d9602e8417d444b828c6401632bffed P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:51.108603 6621 tablet_bootstrap.cc:492] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:51.110208 6623 tablet_bootstrap.cc:492] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:51.110355 6621 tablet_bootstrap.cc:654] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:51.110507 6623 ts_tablet_manager.cc:1367] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.005s
I20240501 20:09:51.111841 6623 raft_consensus.cc:348] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:51.112265 6623 raft_consensus.cc:374] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:51.112537 6623 raft_consensus.cc:729] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:51.113023 6623 consensus_queue.cc:260] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:51.114095 6623 ts_tablet_manager.cc:1398] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:51.116891 6623 tablet_bootstrap.cc:492] T db31c514cd704786b4be2217daaa11d5 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:51.122879 6623 tablet_bootstrap.cc:654] T db31c514cd704786b4be2217daaa11d5 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:51.129894 6619 tablet_bootstrap.cc:492] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:51.130321 6619 ts_tablet_manager.cc:1367] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.000s
I20240501 20:09:51.130815 6621 tablet_bootstrap.cc:492] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:51.130976 6621 ts_tablet_manager.cc:1367] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.004s
I20240501 20:09:51.130990 6620 tablet_bootstrap.cc:492] T 5d9602e8417d444b828c6401632bffed P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:51.131148 6620 ts_tablet_manager.cc:1367] T 5d9602e8417d444b828c6401632bffed P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.030s user 0.015s sys 0.008s
I20240501 20:09:51.131484 6619 raft_consensus.cc:348] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:51.131780 6619 raft_consensus.cc:374] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:51.131781 6621 raft_consensus.cc:348] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:51.132001 6621 raft_consensus.cc:374] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:51.131923 6619 raft_consensus.cc:729] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:51.132100 6621 raft_consensus.cc:729] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:51.132012 6620 raft_consensus.cc:348] T 5d9602e8417d444b828c6401632bffed P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:51.132206 6620 raft_consensus.cc:374] T 5d9602e8417d444b828c6401632bffed P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:51.132349 6620 raft_consensus.cc:729] T 5d9602e8417d444b828c6401632bffed P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:51.132321 6621 consensus_queue.cc:260] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:51.132406 6619 consensus_queue.cc:260] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:51.132676 6620 consensus_queue.cc:260] T 5d9602e8417d444b828c6401632bffed P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:51.133227 6619 ts_tablet_manager.cc:1398] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:51.133328 6620 ts_tablet_manager.cc:1398] T 5d9602e8417d444b828c6401632bffed P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:51.134166 6621 ts_tablet_manager.cc:1398] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:51.134487 6619 tablet_bootstrap.cc:492] T 66c533992f8b4c7a937e79f397636bcc P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:51.134791 6620 tablet_bootstrap.cc:492] T db31c514cd704786b4be2217daaa11d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:51.136536 6620 tablet_bootstrap.cc:654] T db31c514cd704786b4be2217daaa11d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:51.138916 6620 tablet_bootstrap.cc:492] T db31c514cd704786b4be2217daaa11d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:51.139163 6620 ts_tablet_manager.cc:1367] T db31c514cd704786b4be2217daaa11d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240501 20:09:51.140257 6623 tablet_bootstrap.cc:492] T db31c514cd704786b4be2217daaa11d5 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:51.140445 6623 ts_tablet_manager.cc:1367] T db31c514cd704786b4be2217daaa11d5 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.002s
I20240501 20:09:51.140187 6619 tablet_bootstrap.cc:654] T 66c533992f8b4c7a937e79f397636bcc P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:51.141304 6623 raft_consensus.cc:348] T db31c514cd704786b4be2217daaa11d5 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:51.141541 6623 raft_consensus.cc:374] T db31c514cd704786b4be2217daaa11d5 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:51.141681 6623 raft_consensus.cc:729] T db31c514cd704786b4be2217daaa11d5 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:51.141921 6620 raft_consensus.cc:348] T db31c514cd704786b4be2217daaa11d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:51.142220 6620 raft_consensus.cc:374] T db31c514cd704786b4be2217daaa11d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:51.142349 6620 raft_consensus.cc:729] T db31c514cd704786b4be2217daaa11d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:51.142066 6623 consensus_queue.cc:260] T db31c514cd704786b4be2217daaa11d5 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:51.142705 6620 consensus_queue.cc:260] T db31c514cd704786b4be2217daaa11d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:51.143380 6620 ts_tablet_manager.cc:1398] T db31c514cd704786b4be2217daaa11d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 20:09:51.144707 6623 ts_tablet_manager.cc:1398] T db31c514cd704786b4be2217daaa11d5 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:09:51.145546 6619 tablet_bootstrap.cc:492] T 66c533992f8b4c7a937e79f397636bcc P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:51.145793 6619 ts_tablet_manager.cc:1367] T 66c533992f8b4c7a937e79f397636bcc P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20240501 20:09:51.146693 6619 raft_consensus.cc:348] T 66c533992f8b4c7a937e79f397636bcc P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:51.146992 6619 raft_consensus.cc:374] T 66c533992f8b4c7a937e79f397636bcc P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:51.147114 6619 raft_consensus.cc:729] T 66c533992f8b4c7a937e79f397636bcc P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:51.147401 6619 consensus_queue.cc:260] T 66c533992f8b4c7a937e79f397636bcc P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:51.147982 6619 ts_tablet_manager.cc:1398] T 66c533992f8b4c7a937e79f397636bcc P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:51.158149 6614 raft_consensus.cc:482] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:51.158331 6614 raft_consensus.cc:504] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:51.159082 6614 leader_election.cc:288] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:51.159531 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db31c514cd704786b4be2217daaa11d5" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:51.159648 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db31c514cd704786b4be2217daaa11d5" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:51.159761 4498 raft_consensus.cc:2454] T db31c514cd704786b4be2217daaa11d5 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:51.159897 4631 raft_consensus.cc:2454] T db31c514cd704786b4be2217daaa11d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:51.160245 4169 leader_election.cc:302] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:51.160590 6614 raft_consensus.cc:2781] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:51.160681 6614 raft_consensus.cc:482] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:51.160755 6614 raft_consensus.cc:3037] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:51.163156 6614 raft_consensus.cc:504] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:51.163653 6614 leader_election.cc:288] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:51.164253 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db31c514cd704786b4be2217daaa11d5" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:51.164466 4631 raft_consensus.cc:3037] T db31c514cd704786b4be2217daaa11d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:51.164414 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db31c514cd704786b4be2217daaa11d5" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:51.164625 4498 raft_consensus.cc:3037] T db31c514cd704786b4be2217daaa11d5 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:51.167131 4498 raft_consensus.cc:2454] T db31c514cd704786b4be2217daaa11d5 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:51.167224 4631 raft_consensus.cc:2454] T db31c514cd704786b4be2217daaa11d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:51.167702 4169 leader_election.cc:302] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:51.168092 6614 raft_consensus.cc:2781] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:51.168247 6614 raft_consensus.cc:686] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:09:51.168524 6614 consensus_queue.cc:237] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:51.172334 4083 catalog_manager.cc:5484] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:51.293463 6395 raft_consensus.cc:482] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:51.293642 6395 raft_consensus.cc:504] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:51.294348 6395 leader_election.cc:288] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:51.294817 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c533992f8b4c7a937e79f397636bcc" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:51.295070 4235 raft_consensus.cc:2454] T 66c533992f8b4c7a937e79f397636bcc P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:51.295068 4366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c533992f8b4c7a937e79f397636bcc" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:51.295305 4366 raft_consensus.cc:2454] T 66c533992f8b4c7a937e79f397636bcc P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:51.295627 4434 leader_election.cc:302] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:51.295938 6395 raft_consensus.cc:2781] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:51.296041 6395 raft_consensus.cc:482] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:51.296149 6395 raft_consensus.cc:3037] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:51.298429 6395 raft_consensus.cc:504] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:51.299070 6395 leader_election.cc:288] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Requested vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:51.300050 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c533992f8b4c7a937e79f397636bcc" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:51.300299 4235 raft_consensus.cc:3037] T 66c533992f8b4c7a937e79f397636bcc P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:51.302872 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c533992f8b4c7a937e79f397636bcc" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:51.303072 4365 raft_consensus.cc:3037] T 66c533992f8b4c7a937e79f397636bcc P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:51.303215 4235 raft_consensus.cc:2454] T 66c533992f8b4c7a937e79f397636bcc P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:51.303594 4434 leader_election.cc:302] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:51.303905 6395 raft_consensus.cc:2781] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:51.304108 6395 raft_consensus.cc:686] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:09:51.304374 6395 consensus_queue.cc:237] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:51.305184 4365 raft_consensus.cc:2454] T 66c533992f8b4c7a937e79f397636bcc P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:51.307754 4083 catalog_manager.cc:5484] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:51.326704 6557 raft_consensus.cc:482] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:51.326896 6557 raft_consensus.cc:504] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:51.327597 6557 leader_election.cc:288] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:09:51.327948 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d9602e8417d444b828c6401632bffed" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:51.328193 4631 raft_consensus.cc:2454] T 5d9602e8417d444b828c6401632bffed P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:09:51.328325 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d9602e8417d444b828c6401632bffed" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:51.328632 4235 raft_consensus.cc:2454] T 5d9602e8417d444b828c6401632bffed P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:09:51.328703 4302 leader_election.cc:302] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 1d34e18a1d394d3f91c9f50d1c7bea4e; no voters:
I20240501 20:09:51.329851 6634 raft_consensus.cc:2781] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:51.329967 6634 raft_consensus.cc:482] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:51.330083 6634 raft_consensus.cc:3037] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:51.332378 6634 raft_consensus.cc:504] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:51.332958 6634 leader_election.cc:288] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:09:51.333424 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d9602e8417d444b828c6401632bffed" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:51.333555 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d9602e8417d444b828c6401632bffed" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:51.333686 4631 raft_consensus.cc:3037] T 5d9602e8417d444b828c6401632bffed P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:51.333751 4235 raft_consensus.cc:3037] T 5d9602e8417d444b828c6401632bffed P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:51.336052 4631 raft_consensus.cc:2454] T 5d9602e8417d444b828c6401632bffed P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:09:51.336055 4235 raft_consensus.cc:2454] T 5d9602e8417d444b828c6401632bffed P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:09:51.336449 4302 leader_election.cc:302] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 1d34e18a1d394d3f91c9f50d1c7bea4e; no voters:
I20240501 20:09:51.336735 6634 raft_consensus.cc:2781] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:51.336941 6634 raft_consensus.cc:686] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Becoming Leader. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Running, Role: LEADER
I20240501 20:09:51.337234 6634 consensus_queue.cc:237] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:51.340721 4083 catalog_manager.cc:5484] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a reported cstate change: term changed from 0 to 1, leader changed from <none> to 10b47782558c41ee91f5e164a372049a (127.29.219.66). New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:51.609740 6614 consensus_queue.cc:1035] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:51.616036 6614 consensus_queue.cc:1035] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:51.743948 6395 consensus_queue.cc:1035] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:51.750723 6636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:51.751173 6636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 20:09:51.752460 6395 consensus_queue.cc:1035] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:51.753578 6634 consensus_queue.cc:1035] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:51.765262 6636 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:09:51.771545 6474 consensus_queue.cc:1035] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_16'
INSERT report
rows total: 2000
time total: 254.747 ms
time per row: 0.127374 ms
I20240501 20:09:52.196687 4092 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:52.207494 4348 tablet_service.cc:1467] Processing CreateTablet for tablet a20287c2cab44a62b28037a463d4b49b (DEFAULT_TABLE table=rebalance_test_table_17 [id=09dd73eba68c4f8499a786750d39c1cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:52.207909 4348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a20287c2cab44a62b28037a463d4b49b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:52.209841 4479 tablet_service.cc:1467] Processing CreateTablet for tablet a20287c2cab44a62b28037a463d4b49b (DEFAULT_TABLE table=rebalance_test_table_17 [id=09dd73eba68c4f8499a786750d39c1cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:52.210253 4479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a20287c2cab44a62b28037a463d4b49b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:52.211557 4610 tablet_service.cc:1467] Processing CreateTablet for tablet a20287c2cab44a62b28037a463d4b49b (DEFAULT_TABLE table=rebalance_test_table_17 [id=09dd73eba68c4f8499a786750d39c1cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:52.211946 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a20287c2cab44a62b28037a463d4b49b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:52.215018 4611 tablet_service.cc:1467] Processing CreateTablet for tablet ffaf0a0105ea4e1f9ed9d90c221ebc8a (DEFAULT_TABLE table=rebalance_test_table_17 [id=09dd73eba68c4f8499a786750d39c1cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:52.215083 4478 tablet_service.cc:1467] Processing CreateTablet for tablet ffaf0a0105ea4e1f9ed9d90c221ebc8a (DEFAULT_TABLE table=rebalance_test_table_17 [id=09dd73eba68c4f8499a786750d39c1cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:52.215384 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffaf0a0105ea4e1f9ed9d90c221ebc8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:52.215446 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffaf0a0105ea4e1f9ed9d90c221ebc8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:52.219130 4477 tablet_service.cc:1467] Processing CreateTablet for tablet bb5700eb47e044c182c33183ebb27908 (DEFAULT_TABLE table=rebalance_test_table_17 [id=09dd73eba68c4f8499a786750d39c1cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:52.219537 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb5700eb47e044c182c33183ebb27908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:52.223104 4213 tablet_service.cc:1467] Processing CreateTablet for tablet bb5700eb47e044c182c33183ebb27908 (DEFAULT_TABLE table=rebalance_test_table_17 [id=09dd73eba68c4f8499a786750d39c1cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:52.223522 4213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb5700eb47e044c182c33183ebb27908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:52.225697 4346 tablet_service.cc:1467] Processing CreateTablet for tablet bb5700eb47e044c182c33183ebb27908 (DEFAULT_TABLE table=rebalance_test_table_17 [id=09dd73eba68c4f8499a786750d39c1cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:52.226182 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb5700eb47e044c182c33183ebb27908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:52.226214 4347 tablet_service.cc:1467] Processing CreateTablet for tablet ffaf0a0105ea4e1f9ed9d90c221ebc8a (DEFAULT_TABLE table=rebalance_test_table_17 [id=09dd73eba68c4f8499a786750d39c1cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:52.226583 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffaf0a0105ea4e1f9ed9d90c221ebc8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:52.243783 6707 tablet_bootstrap.cc:492] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:52.244890 6708 tablet_bootstrap.cc:492] T bb5700eb47e044c182c33183ebb27908 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:52.245160 6705 tablet_bootstrap.cc:492] T a20287c2cab44a62b28037a463d4b49b P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:52.246841 6707 tablet_bootstrap.cc:654] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:52.247056 6705 tablet_bootstrap.cc:654] T a20287c2cab44a62b28037a463d4b49b P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:52.248237 6706 tablet_bootstrap.cc:492] T a20287c2cab44a62b28037a463d4b49b P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:52.248597 6708 tablet_bootstrap.cc:654] T bb5700eb47e044c182c33183ebb27908 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:52.252538 6706 tablet_bootstrap.cc:654] T a20287c2cab44a62b28037a463d4b49b P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:52.252629 6708 tablet_bootstrap.cc:492] T bb5700eb47e044c182c33183ebb27908 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:52.252879 6708 ts_tablet_manager.cc:1367] T bb5700eb47e044c182c33183ebb27908 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240501 20:09:52.253778 6708 raft_consensus.cc:348] T bb5700eb47e044c182c33183ebb27908 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:52.254129 6708 raft_consensus.cc:374] T bb5700eb47e044c182c33183ebb27908 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:52.254271 6708 raft_consensus.cc:729] T bb5700eb47e044c182c33183ebb27908 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:52.254336 6705 tablet_bootstrap.cc:492] T a20287c2cab44a62b28037a463d4b49b P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:52.254544 6705 ts_tablet_manager.cc:1367] T a20287c2cab44a62b28037a463d4b49b P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240501 20:09:52.255142 6708 consensus_queue.cc:260] T bb5700eb47e044c182c33183ebb27908 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:52.255652 6705 raft_consensus.cc:348] T a20287c2cab44a62b28037a463d4b49b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:52.256011 6705 raft_consensus.cc:374] T a20287c2cab44a62b28037a463d4b49b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:52.256158 6705 raft_consensus.cc:729] T a20287c2cab44a62b28037a463d4b49b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:52.256253 6708 ts_tablet_manager.cc:1398] T bb5700eb47e044c182c33183ebb27908 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:52.256472 6705 consensus_queue.cc:260] T a20287c2cab44a62b28037a463d4b49b P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:52.256701 6707 tablet_bootstrap.cc:492] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:52.256949 6707 ts_tablet_manager.cc:1367] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.004s
I20240501 20:09:52.257195 6705 ts_tablet_manager.cc:1398] T a20287c2cab44a62b28037a463d4b49b P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:52.257731 6705 tablet_bootstrap.cc:492] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:52.257820 6707 raft_consensus.cc:348] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:52.258097 6707 raft_consensus.cc:374] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:52.258239 6707 raft_consensus.cc:729] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:52.258502 6707 consensus_queue.cc:260] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:52.259588 6705 tablet_bootstrap.cc:654] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:52.262785 6707 ts_tablet_manager.cc:1398] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:09:52.263120 6706 tablet_bootstrap.cc:492] T a20287c2cab44a62b28037a463d4b49b P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:52.263144 6707 tablet_bootstrap.cc:492] T bb5700eb47e044c182c33183ebb27908 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:52.263397 6706 ts_tablet_manager.cc:1367] T a20287c2cab44a62b28037a463d4b49b P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20240501 20:09:52.264273 6706 raft_consensus.cc:348] T a20287c2cab44a62b28037a463d4b49b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:52.264550 6706 raft_consensus.cc:374] T a20287c2cab44a62b28037a463d4b49b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:52.264676 6706 raft_consensus.cc:729] T a20287c2cab44a62b28037a463d4b49b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:52.264758 6707 tablet_bootstrap.cc:654] T bb5700eb47e044c182c33183ebb27908 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:52.264940 6706 consensus_queue.cc:260] T a20287c2cab44a62b28037a463d4b49b P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:52.265648 6706 ts_tablet_manager.cc:1398] T a20287c2cab44a62b28037a463d4b49b P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:52.265933 6706 tablet_bootstrap.cc:492] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:52.267508 6706 tablet_bootstrap.cc:654] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:52.276396 6707 tablet_bootstrap.cc:492] T bb5700eb47e044c182c33183ebb27908 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:52.276646 6707 ts_tablet_manager.cc:1367] T bb5700eb47e044c182c33183ebb27908 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.004s
I20240501 20:09:52.277793 6707 raft_consensus.cc:348] T bb5700eb47e044c182c33183ebb27908 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:52.278221 6707 raft_consensus.cc:374] T bb5700eb47e044c182c33183ebb27908 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:52.278332 6707 raft_consensus.cc:729] T bb5700eb47e044c182c33183ebb27908 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:52.278672 6707 consensus_queue.cc:260] T bb5700eb47e044c182c33183ebb27908 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:52.279953 6705 tablet_bootstrap.cc:492] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:52.280179 6705 ts_tablet_manager.cc:1367] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20240501 20:09:52.281291 6707 ts_tablet_manager.cc:1398] T bb5700eb47e044c182c33183ebb27908 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240501 20:09:52.281751 6707 tablet_bootstrap.cc:492] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:52.283485 6705 raft_consensus.cc:348] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:52.283777 6705 raft_consensus.cc:374] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:52.283876 6705 raft_consensus.cc:729] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:52.284166 6705 consensus_queue.cc:260] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:52.285779 6707 tablet_bootstrap.cc:654] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:52.289353 6705 ts_tablet_manager.cc:1398] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240501 20:09:52.292469 6705 tablet_bootstrap.cc:492] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:52.294108 6705 tablet_bootstrap.cc:654] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:52.300525 6707 tablet_bootstrap.cc:492] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:52.300700 6707 ts_tablet_manager.cc:1367] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.006s
I20240501 20:09:52.300817 6706 tablet_bootstrap.cc:492] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:52.300983 6706 ts_tablet_manager.cc:1367] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.020s
I20240501 20:09:52.301523 6707 raft_consensus.cc:348] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:52.301752 6707 raft_consensus.cc:374] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:52.301733 6706 raft_consensus.cc:348] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:52.301862 6707 raft_consensus.cc:729] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:52.301954 6706 raft_consensus.cc:374] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:52.302037 6706 raft_consensus.cc:729] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:52.302125 6707 consensus_queue.cc:260] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:52.302260 6706 consensus_queue.cc:260] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:52.302805 6706 ts_tablet_manager.cc:1398] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:09:52.302920 6705 tablet_bootstrap.cc:492] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:52.303146 6705 ts_tablet_manager.cc:1367] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.003s
I20240501 20:09:52.303432 6707 ts_tablet_manager.cc:1398] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:09:52.303951 6705 raft_consensus.cc:348] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:52.304201 6705 raft_consensus.cc:374] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:52.304344 6705 raft_consensus.cc:729] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:52.304641 6705 consensus_queue.cc:260] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:52.305271 6705 ts_tablet_manager.cc:1398] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:09:52.543643 6395 raft_consensus.cc:482] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:52.543819 6395 raft_consensus.cc:504] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:52.544473 6395 leader_election.cc:288] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:52.544955 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffaf0a0105ea4e1f9ed9d90c221ebc8a" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:52.545032 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffaf0a0105ea4e1f9ed9d90c221ebc8a" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:52.545218 4631 raft_consensus.cc:2454] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:52.545240 4365 raft_consensus.cc:2454] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:52.545694 4434 leader_election.cc:302] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:52.545975 6395 raft_consensus.cc:2781] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:52.546100 6395 raft_consensus.cc:482] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:52.546208 6395 raft_consensus.cc:3037] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:52.548575 6395 raft_consensus.cc:504] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:52.549165 6395 leader_election.cc:288] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:52.549582 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffaf0a0105ea4e1f9ed9d90c221ebc8a" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:52.549665 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffaf0a0105ea4e1f9ed9d90c221ebc8a" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:52.549755 4631 raft_consensus.cc:3037] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:52.549922 4365 raft_consensus.cc:3037] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:52.552182 4365 raft_consensus.cc:2454] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:52.552327 4631 raft_consensus.cc:2454] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:52.552634 4434 leader_election.cc:302] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:52.553098 6395 raft_consensus.cc:2781] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:52.553447 6395 raft_consensus.cc:686] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:09:52.553771 6395 consensus_queue.cc:237] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:52.557338 4092 catalog_manager.cc:5484] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:52.584046 6395 raft_consensus.cc:482] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:52.584236 6395 raft_consensus.cc:504] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:52.584888 6395 leader_election.cc:288] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:52.585358 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20287c2cab44a62b28037a463d4b49b" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:52.585319 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20287c2cab44a62b28037a463d4b49b" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:52.585580 4365 raft_consensus.cc:2454] T a20287c2cab44a62b28037a463d4b49b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:52.585682 4631 raft_consensus.cc:2454] T a20287c2cab44a62b28037a463d4b49b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:52.586028 4434 leader_election.cc:302] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:52.586454 6395 raft_consensus.cc:2781] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:52.586558 6395 raft_consensus.cc:482] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:52.586673 6395 raft_consensus.cc:3037] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:52.589393 6395 raft_consensus.cc:504] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:52.589944 6395 leader_election.cc:288] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:52.590389 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20287c2cab44a62b28037a463d4b49b" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:52.590533 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a20287c2cab44a62b28037a463d4b49b" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:52.590603 4365 raft_consensus.cc:3037] T a20287c2cab44a62b28037a463d4b49b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:52.590764 4631 raft_consensus.cc:3037] T a20287c2cab44a62b28037a463d4b49b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:52.593560 4631 raft_consensus.cc:2454] T a20287c2cab44a62b28037a463d4b49b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:52.593573 4365 raft_consensus.cc:2454] T a20287c2cab44a62b28037a463d4b49b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:52.594007 4435 leader_election.cc:302] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:52.594290 6395 raft_consensus.cc:2781] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:52.594455 6395 raft_consensus.cc:686] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:09:52.595261 6395 consensus_queue.cc:237] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:09:52.598958 4092 catalog_manager.cc:5484] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:52.618927 6557 raft_consensus.cc:482] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:52.619115 6557 raft_consensus.cc:504] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:52.619627 6557 leader_election.cc:288] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:09:52.620105 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5700eb47e044c182c33183ebb27908" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:52.620162 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5700eb47e044c182c33183ebb27908" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:52.620316 4234 raft_consensus.cc:2454] T bb5700eb47e044c182c33183ebb27908 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:09:52.620339 4499 raft_consensus.cc:2454] T bb5700eb47e044c182c33183ebb27908 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:09:52.620718 4301 leader_election.cc:302] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:52.621002 6557 raft_consensus.cc:2781] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:52.621085 6557 raft_consensus.cc:482] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:52.621188 6557 raft_consensus.cc:3037] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:52.623183 6557 raft_consensus.cc:504] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:52.623718 6557 leader_election.cc:288] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Requested vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:09:52.624178 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5700eb47e044c182c33183ebb27908" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:52.624274 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb5700eb47e044c182c33183ebb27908" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:52.624426 4234 raft_consensus.cc:3037] T bb5700eb47e044c182c33183ebb27908 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:52.624466 4499 raft_consensus.cc:3037] T bb5700eb47e044c182c33183ebb27908 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:52.627240 4499 raft_consensus.cc:2454] T bb5700eb47e044c182c33183ebb27908 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:09:52.627224 4234 raft_consensus.cc:2454] T bb5700eb47e044c182c33183ebb27908 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:09:52.627704 4301 leader_election.cc:302] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:52.628002 6557 raft_consensus.cc:2781] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:52.628268 6557 raft_consensus.cc:686] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Becoming Leader. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Running, Role: LEADER
I20240501 20:09:52.628535 6557 consensus_queue.cc:237] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:52.633610 4092 catalog_manager.cc:5484] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a reported cstate change: term changed from 0 to 1, leader changed from <none> to 10b47782558c41ee91f5e164a372049a (127.29.219.66). New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:52.987370 6395 consensus_queue.cc:1035] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:52.994052 6395 consensus_queue.cc:1035] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:53.034657 6395 consensus_queue.cc:1035] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:53.040005 6395 consensus_queue.cc:1035] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:53.061215 6721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:53.061532 6721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:53.069419 6721 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_17'
I20240501 20:09:53.151588 6634 consensus_queue.cc:1035] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:53.212059 6557 consensus_queue.cc:1035] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 299.862 ms
time per row: 0.149931 ms
I20240501 20:09:53.534276 4085 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:53.543017 4213 tablet_service.cc:1467] Processing CreateTablet for tablet 80d18632bc0443558294b9f99bf08654 (DEFAULT_TABLE table=rebalance_test_table_18 [id=b5dc7891d8a1410bb1280ac3ce2fd364]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:53.543174 4348 tablet_service.cc:1467] Processing CreateTablet for tablet 80d18632bc0443558294b9f99bf08654 (DEFAULT_TABLE table=rebalance_test_table_18 [id=b5dc7891d8a1410bb1280ac3ce2fd364]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:53.543501 4213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d18632bc0443558294b9f99bf08654. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.543632 4348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d18632bc0443558294b9f99bf08654. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.543602 4478 tablet_service.cc:1467] Processing CreateTablet for tablet c2c09fbc593b4fb5a97e05f5c8964522 (DEFAULT_TABLE table=rebalance_test_table_18 [id=b5dc7891d8a1410bb1280ac3ce2fd364]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:53.543989 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2c09fbc593b4fb5a97e05f5c8964522. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.543979 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 80d18632bc0443558294b9f99bf08654 (DEFAULT_TABLE table=rebalance_test_table_18 [id=b5dc7891d8a1410bb1280ac3ce2fd364]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:53.544392 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d18632bc0443558294b9f99bf08654. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.550704 4609 tablet_service.cc:1467] Processing CreateTablet for tablet cde1b2be0bcd4337baed241933703f42 (DEFAULT_TABLE table=rebalance_test_table_18 [id=b5dc7891d8a1410bb1280ac3ce2fd364]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:53.551190 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cde1b2be0bcd4337baed241933703f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.554278 4346 tablet_service.cc:1467] Processing CreateTablet for tablet cde1b2be0bcd4337baed241933703f42 (DEFAULT_TABLE table=rebalance_test_table_18 [id=b5dc7891d8a1410bb1280ac3ce2fd364]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:53.555485 4611 tablet_service.cc:1467] Processing CreateTablet for tablet c2c09fbc593b4fb5a97e05f5c8964522 (DEFAULT_TABLE table=rebalance_test_table_18 [id=b5dc7891d8a1410bb1280ac3ce2fd364]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:53.555999 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2c09fbc593b4fb5a97e05f5c8964522. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.557708 6780 tablet_bootstrap.cc:492] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:53.560561 4215 tablet_service.cc:1467] Processing CreateTablet for tablet cde1b2be0bcd4337baed241933703f42 (DEFAULT_TABLE table=rebalance_test_table_18 [id=b5dc7891d8a1410bb1280ac3ce2fd364]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:53.560604 4214 tablet_service.cc:1467] Processing CreateTablet for tablet c2c09fbc593b4fb5a97e05f5c8964522 (DEFAULT_TABLE table=rebalance_test_table_18 [id=b5dc7891d8a1410bb1280ac3ce2fd364]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:53.561079 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cde1b2be0bcd4337baed241933703f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.562412 6780 tablet_bootstrap.cc:654] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.562665 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2c09fbc593b4fb5a97e05f5c8964522. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.565265 6782 tablet_bootstrap.cc:492] T 80d18632bc0443558294b9f99bf08654 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:53.567624 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cde1b2be0bcd4337baed241933703f42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:53.568127 6780 tablet_bootstrap.cc:492] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:53.568269 6780 ts_tablet_manager.cc:1367] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.002s
I20240501 20:09:53.569079 6780 raft_consensus.cc:348] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:53.569291 6780 raft_consensus.cc:374] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:53.569383 6780 raft_consensus.cc:729] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:53.570288 6782 tablet_bootstrap.cc:654] T 80d18632bc0443558294b9f99bf08654 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.569633 6780 consensus_queue.cc:260] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:53.571825 6780 ts_tablet_manager.cc:1398] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:53.589761 6780 tablet_bootstrap.cc:492] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:53.590922 6783 tablet_bootstrap.cc:492] T 80d18632bc0443558294b9f99bf08654 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:53.592875 6780 tablet_bootstrap.cc:654] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.593847 6783 tablet_bootstrap.cc:654] T 80d18632bc0443558294b9f99bf08654 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.596810 6781 tablet_bootstrap.cc:492] T c2c09fbc593b4fb5a97e05f5c8964522 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:53.598800 6781 tablet_bootstrap.cc:654] T c2c09fbc593b4fb5a97e05f5c8964522 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.600291 6782 tablet_bootstrap.cc:492] T 80d18632bc0443558294b9f99bf08654 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:53.600557 6782 ts_tablet_manager.cc:1367] T 80d18632bc0443558294b9f99bf08654 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.035s user 0.012s sys 0.008s
I20240501 20:09:53.601365 6783 tablet_bootstrap.cc:492] T 80d18632bc0443558294b9f99bf08654 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:53.601514 6783 ts_tablet_manager.cc:1367] T 80d18632bc0443558294b9f99bf08654 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240501 20:09:53.601686 6782 raft_consensus.cc:348] T 80d18632bc0443558294b9f99bf08654 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:53.601939 6782 raft_consensus.cc:374] T 80d18632bc0443558294b9f99bf08654 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:53.602049 6782 raft_consensus.cc:729] T 80d18632bc0443558294b9f99bf08654 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:53.602254 6783 raft_consensus.cc:348] T 80d18632bc0443558294b9f99bf08654 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:53.602341 6782 consensus_queue.cc:260] T 80d18632bc0443558294b9f99bf08654 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:53.602685 6783 raft_consensus.cc:374] T 80d18632bc0443558294b9f99bf08654 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:53.602852 6780 tablet_bootstrap.cc:492] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:53.602846 6783 raft_consensus.cc:729] T 80d18632bc0443558294b9f99bf08654 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:53.603049 6780 ts_tablet_manager.cc:1367] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20240501 20:09:53.603214 6781 tablet_bootstrap.cc:492] T c2c09fbc593b4fb5a97e05f5c8964522 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:53.603418 6781 ts_tablet_manager.cc:1367] T c2c09fbc593b4fb5a97e05f5c8964522 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20240501 20:09:53.603369 6783 consensus_queue.cc:260] T 80d18632bc0443558294b9f99bf08654 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:53.604573 6782 ts_tablet_manager.cc:1398] T 80d18632bc0443558294b9f99bf08654 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 20:09:53.604872 6782 tablet_bootstrap.cc:492] T cde1b2be0bcd4337baed241933703f42 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:53.606295 6781 raft_consensus.cc:348] T c2c09fbc593b4fb5a97e05f5c8964522 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:53.606467 6782 tablet_bootstrap.cc:654] T cde1b2be0bcd4337baed241933703f42 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.606596 6781 raft_consensus.cc:374] T c2c09fbc593b4fb5a97e05f5c8964522 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:53.605253 6780 raft_consensus.cc:348] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:53.606772 6781 raft_consensus.cc:729] T c2c09fbc593b4fb5a97e05f5c8964522 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:53.606930 6780 raft_consensus.cc:374] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:53.607045 6780 raft_consensus.cc:729] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:53.607141 6781 consensus_queue.cc:260] T c2c09fbc593b4fb5a97e05f5c8964522 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:53.607331 6780 consensus_queue.cc:260] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:53.608047 6780 ts_tablet_manager.cc:1398] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:09:53.609292 6781 ts_tablet_manager.cc:1398] T c2c09fbc593b4fb5a97e05f5c8964522 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 20:09:53.613746 6783 ts_tablet_manager.cc:1398] T 80d18632bc0443558294b9f99bf08654 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 20:09:53.616248 6783 tablet_bootstrap.cc:492] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:53.616811 6782 tablet_bootstrap.cc:492] T cde1b2be0bcd4337baed241933703f42 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:53.616991 6782 ts_tablet_manager.cc:1367] T cde1b2be0bcd4337baed241933703f42 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.007s
I20240501 20:09:53.617790 6782 raft_consensus.cc:348] T cde1b2be0bcd4337baed241933703f42 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:53.618028 6782 raft_consensus.cc:374] T cde1b2be0bcd4337baed241933703f42 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:53.618110 6782 raft_consensus.cc:729] T cde1b2be0bcd4337baed241933703f42 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:53.618369 6783 tablet_bootstrap.cc:654] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.618383 6782 consensus_queue.cc:260] T cde1b2be0bcd4337baed241933703f42 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:53.619051 6782 ts_tablet_manager.cc:1398] T cde1b2be0bcd4337baed241933703f42 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 20:09:53.619552 6782 tablet_bootstrap.cc:492] T c2c09fbc593b4fb5a97e05f5c8964522 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:53.621466 6782 tablet_bootstrap.cc:654] T c2c09fbc593b4fb5a97e05f5c8964522 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.627905 6783 tablet_bootstrap.cc:492] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:53.628121 6782 tablet_bootstrap.cc:492] T c2c09fbc593b4fb5a97e05f5c8964522 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:53.628190 6783 ts_tablet_manager.cc:1367] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20240501 20:09:53.628276 6782 ts_tablet_manager.cc:1367] T c2c09fbc593b4fb5a97e05f5c8964522 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240501 20:09:53.628890 6783 raft_consensus.cc:348] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:53.629129 6783 raft_consensus.cc:374] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:53.629204 6783 raft_consensus.cc:729] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:53.629106 6782 raft_consensus.cc:348] T c2c09fbc593b4fb5a97e05f5c8964522 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:53.629302 6782 raft_consensus.cc:374] T c2c09fbc593b4fb5a97e05f5c8964522 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:53.629396 6782 raft_consensus.cc:729] T c2c09fbc593b4fb5a97e05f5c8964522 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:53.629448 6783 consensus_queue.cc:260] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:53.629633 6782 consensus_queue.cc:260] T c2c09fbc593b4fb5a97e05f5c8964522 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:53.630051 6783 ts_tablet_manager.cc:1398] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:09:53.630590 6783 tablet_bootstrap.cc:492] T cde1b2be0bcd4337baed241933703f42 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:53.631368 6782 ts_tablet_manager.cc:1398] T c2c09fbc593b4fb5a97e05f5c8964522 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:09:53.632378 6783 tablet_bootstrap.cc:654] T cde1b2be0bcd4337baed241933703f42 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:53.634173 6783 tablet_bootstrap.cc:492] T cde1b2be0bcd4337baed241933703f42 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:53.634315 6783 ts_tablet_manager.cc:1367] T cde1b2be0bcd4337baed241933703f42 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 20:09:53.635097 6783 raft_consensus.cc:348] T cde1b2be0bcd4337baed241933703f42 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:53.635318 6783 raft_consensus.cc:374] T cde1b2be0bcd4337baed241933703f42 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:53.635407 6783 raft_consensus.cc:729] T cde1b2be0bcd4337baed241933703f42 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:53.635661 6783 consensus_queue.cc:260] T cde1b2be0bcd4337baed241933703f42 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:53.636327 6783 ts_tablet_manager.cc:1398] T cde1b2be0bcd4337baed241933703f42 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:53.638154 6639 raft_consensus.cc:482] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:53.638269 6639 raft_consensus.cc:504] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:53.638896 6639 leader_election.cc:288] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:53.639223 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c09fbc593b4fb5a97e05f5c8964522" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:09:53.639461 4499 raft_consensus.cc:2454] T c2c09fbc593b4fb5a97e05f5c8964522 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:53.639492 4629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c09fbc593b4fb5a97e05f5c8964522" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:53.639698 4629 raft_consensus.cc:2454] T c2c09fbc593b4fb5a97e05f5c8964522 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:09:53.640133 4170 leader_election.cc:302] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:53.640446 6639 raft_consensus.cc:2781] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:53.640566 6639 raft_consensus.cc:482] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:53.640652 6639 raft_consensus.cc:3037] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:53.642868 6639 raft_consensus.cc:504] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:53.643641 6639 leader_election.cc:288] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:53.643832 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c09fbc593b4fb5a97e05f5c8964522" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:09:53.644069 4499 raft_consensus.cc:3037] T c2c09fbc593b4fb5a97e05f5c8964522 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:53.643996 4629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2c09fbc593b4fb5a97e05f5c8964522" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:53.644207 4629 raft_consensus.cc:3037] T c2c09fbc593b4fb5a97e05f5c8964522 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:53.647128 4499 raft_consensus.cc:2454] T c2c09fbc593b4fb5a97e05f5c8964522 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:53.647146 4629 raft_consensus.cc:2454] T c2c09fbc593b4fb5a97e05f5c8964522 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:09:53.647555 4170 leader_election.cc:302] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:53.647830 6639 raft_consensus.cc:2781] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:53.648090 6639 raft_consensus.cc:686] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:09:53.648423 6639 consensus_queue.cc:237] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:53.653391 4092 catalog_manager.cc:5484] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:53.738423 6634 raft_consensus.cc:482] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:53.738582 6634 raft_consensus.cc:504] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:53.739369 6634 leader_election.cc:288] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:09:53.739751 4629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cde1b2be0bcd4337baed241933703f42" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:53.739866 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cde1b2be0bcd4337baed241933703f42" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:53.739984 4629 raft_consensus.cc:2454] T cde1b2be0bcd4337baed241933703f42 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:09:53.740106 4234 raft_consensus.cc:2454] T cde1b2be0bcd4337baed241933703f42 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:09:53.740514 4302 leader_election.cc:302] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 1d34e18a1d394d3f91c9f50d1c7bea4e; no voters:
I20240501 20:09:53.740860 6634 raft_consensus.cc:2781] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:53.740973 6634 raft_consensus.cc:482] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:53.741091 6634 raft_consensus.cc:3037] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:53.743100 6634 raft_consensus.cc:504] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:53.743643 6634 leader_election.cc:288] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:09:53.744105 4629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cde1b2be0bcd4337baed241933703f42" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:53.744319 4629 raft_consensus.cc:3037] T cde1b2be0bcd4337baed241933703f42 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:53.744262 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cde1b2be0bcd4337baed241933703f42" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:53.744477 4234 raft_consensus.cc:3037] T cde1b2be0bcd4337baed241933703f42 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:53.746243 4629 raft_consensus.cc:2454] T cde1b2be0bcd4337baed241933703f42 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:09:53.746341 4234 raft_consensus.cc:2454] T cde1b2be0bcd4337baed241933703f42 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:09:53.746729 4302 leader_election.cc:302] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 1d34e18a1d394d3f91c9f50d1c7bea4e; no voters:
I20240501 20:09:53.747013 6634 raft_consensus.cc:2781] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:53.747146 6634 raft_consensus.cc:686] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Becoming Leader. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Running, Role: LEADER
I20240501 20:09:53.747444 6634 consensus_queue.cc:237] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:53.751471 4092 catalog_manager.cc:5484] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a reported cstate change: term changed from 0 to 1, leader changed from <none> to 10b47782558c41ee91f5e164a372049a (127.29.219.66). New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } }
I20240501 20:09:53.804672 6634 raft_consensus.cc:482] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:53.804916 6634 raft_consensus.cc:504] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:53.805486 6634 leader_election.cc:288] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:53.805887 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d18632bc0443558294b9f99bf08654" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:53.805917 4629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d18632bc0443558294b9f99bf08654" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:53.806102 4629 raft_consensus.cc:2454] T 80d18632bc0443558294b9f99bf08654 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:09:53.806097 4234 raft_consensus.cc:2454] T 80d18632bc0443558294b9f99bf08654 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:09:53.806445 4301 leader_election.cc:302] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:53.806773 6634 raft_consensus.cc:2781] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:53.806856 6634 raft_consensus.cc:482] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:53.806929 6634 raft_consensus.cc:3037] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:53.809551 6634 raft_consensus.cc:504] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:53.810168 6634 leader_election.cc:288] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Requested vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:53.810678 4629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d18632bc0443558294b9f99bf08654" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:53.810654 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80d18632bc0443558294b9f99bf08654" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:53.810886 4629 raft_consensus.cc:3037] T 80d18632bc0443558294b9f99bf08654 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:53.810905 4234 raft_consensus.cc:3037] T 80d18632bc0443558294b9f99bf08654 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:53.813737 4234 raft_consensus.cc:2454] T 80d18632bc0443558294b9f99bf08654 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:09:53.813753 4629 raft_consensus.cc:2454] T 80d18632bc0443558294b9f99bf08654 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:09:53.814162 4302 leader_election.cc:302] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 1d34e18a1d394d3f91c9f50d1c7bea4e; no voters:
I20240501 20:09:53.814417 6634 raft_consensus.cc:2781] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:53.814600 6634 raft_consensus.cc:686] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Becoming Leader. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Running, Role: LEADER
I20240501 20:09:53.814951 6634 consensus_queue.cc:237] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:09:53.818495 4092 catalog_manager.cc:5484] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a reported cstate change: term changed from 0 to 1, leader changed from <none> to 10b47782558c41ee91f5e164a372049a (127.29.219.66). New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:54.109889 6745 consensus_queue.cc:1035] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:54.126428 6745 consensus_queue.cc:1035] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:54.131008 6557 consensus_queue.cc:1035] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:54.143505 6634 consensus_queue.cc:1035] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:54.246330 6796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:54.246712 6796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:54.256120 6796 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_18'
I20240501 20:09:54.393695 6806 consensus_queue.cc:1035] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:54.416165 6634 consensus_queue.cc:1035] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 279.619 ms
time per row: 0.13981 ms
I20240501 20:09:54.706872 4087 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:54.717630 4478 tablet_service.cc:1467] Processing CreateTablet for tablet ca4f195a6dd14be981395538c2a33ad9 (DEFAULT_TABLE table=rebalance_test_table_19 [id=8de1e355c457486da8bba4c628d52c17]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:54.718101 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4f195a6dd14be981395538c2a33ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:54.719337 4348 tablet_service.cc:1467] Processing CreateTablet for tablet ca4f195a6dd14be981395538c2a33ad9 (DEFAULT_TABLE table=rebalance_test_table_19 [id=8de1e355c457486da8bba4c628d52c17]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:54.719830 4348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4f195a6dd14be981395538c2a33ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:54.723631 4215 tablet_service.cc:1467] Processing CreateTablet for tablet ca4f195a6dd14be981395538c2a33ad9 (DEFAULT_TABLE table=rebalance_test_table_19 [id=8de1e355c457486da8bba4c628d52c17]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:54.724072 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4f195a6dd14be981395538c2a33ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:54.727164 4609 tablet_service.cc:1467] Processing CreateTablet for tablet 7bcb854a8f7545cc86323abd69c26c23 (DEFAULT_TABLE table=rebalance_test_table_19 [id=8de1e355c457486da8bba4c628d52c17]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:54.727617 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bcb854a8f7545cc86323abd69c26c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:54.728145 6858 tablet_bootstrap.cc:492] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:54.729549 6858 tablet_bootstrap.cc:654] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:54.731760 4611 tablet_service.cc:1467] Processing CreateTablet for tablet 4f99f5d9d4bf44c39188c8b5d5362b98 (DEFAULT_TABLE table=rebalance_test_table_19 [id=8de1e355c457486da8bba4c628d52c17]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:54.732152 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f99f5d9d4bf44c39188c8b5d5362b98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:54.735754 4477 tablet_service.cc:1467] Processing CreateTablet for tablet 7bcb854a8f7545cc86323abd69c26c23 (DEFAULT_TABLE table=rebalance_test_table_19 [id=8de1e355c457486da8bba4c628d52c17]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:54.735965 4213 tablet_service.cc:1467] Processing CreateTablet for tablet 4f99f5d9d4bf44c39188c8b5d5362b98 (DEFAULT_TABLE table=rebalance_test_table_19 [id=8de1e355c457486da8bba4c628d52c17]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:54.736284 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bcb854a8f7545cc86323abd69c26c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:54.736234 4346 tablet_service.cc:1467] Processing CreateTablet for tablet 7bcb854a8f7545cc86323abd69c26c23 (DEFAULT_TABLE table=rebalance_test_table_19 [id=8de1e355c457486da8bba4c628d52c17]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:54.736369 4213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f99f5d9d4bf44c39188c8b5d5362b98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:54.736613 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bcb854a8f7545cc86323abd69c26c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:54.747561 4478 tablet_service.cc:1467] Processing CreateTablet for tablet 4f99f5d9d4bf44c39188c8b5d5362b98 (DEFAULT_TABLE table=rebalance_test_table_19 [id=8de1e355c457486da8bba4c628d52c17]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:54.748234 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f99f5d9d4bf44c39188c8b5d5362b98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:54.754655 6860 tablet_bootstrap.cc:492] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:54.759619 6860 tablet_bootstrap.cc:654] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:54.768640 6862 tablet_bootstrap.cc:492] T 7bcb854a8f7545cc86323abd69c26c23 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:54.770562 6860 tablet_bootstrap.cc:492] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:54.770570 6862 tablet_bootstrap.cc:654] T 7bcb854a8f7545cc86323abd69c26c23 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:54.770867 6860 ts_tablet_manager.cc:1367] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.001s
I20240501 20:09:54.771970 6860 raft_consensus.cc:348] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:54.772300 6861 tablet_bootstrap.cc:492] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:54.772462 6860 raft_consensus.cc:374] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.773030 6860 raft_consensus.cc:729] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.773424 6860 consensus_queue.cc:260] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:54.774185 6861 tablet_bootstrap.cc:654] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:54.776504 6862 tablet_bootstrap.cc:492] T 7bcb854a8f7545cc86323abd69c26c23 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:54.776712 6862 ts_tablet_manager.cc:1367] T 7bcb854a8f7545cc86323abd69c26c23 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20240501 20:09:54.777608 6862 raft_consensus.cc:348] T 7bcb854a8f7545cc86323abd69c26c23 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:54.777894 6862 raft_consensus.cc:374] T 7bcb854a8f7545cc86323abd69c26c23 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.778053 6862 raft_consensus.cc:729] T 7bcb854a8f7545cc86323abd69c26c23 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.776682 6860 ts_tablet_manager.cc:1398] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240501 20:09:54.778326 6862 consensus_queue.cc:260] T 7bcb854a8f7545cc86323abd69c26c23 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:54.779026 6862 ts_tablet_manager.cc:1398] T 7bcb854a8f7545cc86323abd69c26c23 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:09:54.779353 6862 tablet_bootstrap.cc:492] T ca4f195a6dd14be981395538c2a33ad9 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:09:54.781534 6800 raft_consensus.cc:482] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:54.781708 6800 raft_consensus.cc:504] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:54.782548 6800 leader_election.cc:288] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:09:54.783313 6860 tablet_bootstrap.cc:492] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:09:54.784137 4629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f99f5d9d4bf44c39188c8b5d5362b98" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:54.784519 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f99f5d9d4bf44c39188c8b5d5362b98" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
W20240501 20:09:54.785244 4170 leader_election.cc:341] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:09:54.785612 4169 leader_election.cc:341] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:54.785734 4169 leader_election.cc:302] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dca57823003e4729b769ca12367d7530; no voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b
I20240501 20:09:54.786163 6639 raft_consensus.cc:2726] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:09:54.786391 6858 tablet_bootstrap.cc:492] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:54.786594 6858 ts_tablet_manager.cc:1367] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.059s user 0.015s sys 0.025s
I20240501 20:09:54.786917 6861 tablet_bootstrap.cc:492] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:54.787117 6861 ts_tablet_manager.cc:1367] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240501 20:09:54.787513 6858 raft_consensus.cc:348] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:54.787793 6858 raft_consensus.cc:374] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.787950 6858 raft_consensus.cc:729] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.787921 6861 raft_consensus.cc:348] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:54.788259 6858 consensus_queue.cc:260] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:54.788514 6860 tablet_bootstrap.cc:654] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:54.789893 6858 ts_tablet_manager.cc:1398] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:54.790017 6861 raft_consensus.cc:374] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.790226 6858 tablet_bootstrap.cc:492] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:54.791780 6861 raft_consensus.cc:729] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.791930 6858 tablet_bootstrap.cc:654] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:54.792214 6861 consensus_queue.cc:260] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:54.792821 6861 ts_tablet_manager.cc:1398] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 20:09:54.781168 6862 tablet_bootstrap.cc:654] T ca4f195a6dd14be981395538c2a33ad9 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:54.794047 6861 tablet_bootstrap.cc:492] T 7bcb854a8f7545cc86323abd69c26c23 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:09:54.797202 6861 tablet_bootstrap.cc:654] T 7bcb854a8f7545cc86323abd69c26c23 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:54.801985 6858 tablet_bootstrap.cc:492] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:54.802282 6858 ts_tablet_manager.cc:1367] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240501 20:09:54.803231 6858 raft_consensus.cc:348] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:54.803552 6858 raft_consensus.cc:374] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.803697 6858 raft_consensus.cc:729] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.804404 6858 consensus_queue.cc:260] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:54.805084 6858 ts_tablet_manager.cc:1398] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:54.805460 6858 tablet_bootstrap.cc:492] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:09:54.805855 6860 tablet_bootstrap.cc:492] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:09:54.806092 6860 ts_tablet_manager.cc:1367] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.001s
I20240501 20:09:54.807288 6860 raft_consensus.cc:348] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:54.808143 6860 raft_consensus.cc:374] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.808634 6860 raft_consensus.cc:729] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.809010 6860 consensus_queue.cc:260] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:54.810257 6861 tablet_bootstrap.cc:492] T 7bcb854a8f7545cc86323abd69c26c23 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:09:54.810365 6862 tablet_bootstrap.cc:492] T ca4f195a6dd14be981395538c2a33ad9 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:09:54.810529 6861 ts_tablet_manager.cc:1367] T 7bcb854a8f7545cc86323abd69c26c23 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.005s
I20240501 20:09:54.810571 6858 tablet_bootstrap.cc:654] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:54.810567 6862 ts_tablet_manager.cc:1367] T ca4f195a6dd14be981395538c2a33ad9 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.004s
I20240501 20:09:54.811435 6862 raft_consensus.cc:348] T ca4f195a6dd14be981395538c2a33ad9 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:54.811749 6862 raft_consensus.cc:374] T ca4f195a6dd14be981395538c2a33ad9 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.811872 6862 raft_consensus.cc:729] T ca4f195a6dd14be981395538c2a33ad9 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.811980 6861 raft_consensus.cc:348] T 7bcb854a8f7545cc86323abd69c26c23 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:54.812191 6862 consensus_queue.cc:260] T ca4f195a6dd14be981395538c2a33ad9 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:54.812839 6862 ts_tablet_manager.cc:1398] T ca4f195a6dd14be981395538c2a33ad9 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:54.813012 6860 ts_tablet_manager.cc:1398] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240501 20:09:54.812441 6861 raft_consensus.cc:374] T 7bcb854a8f7545cc86323abd69c26c23 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.816344 6861 raft_consensus.cc:729] T 7bcb854a8f7545cc86323abd69c26c23 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.816751 6861 consensus_queue.cc:260] T 7bcb854a8f7545cc86323abd69c26c23 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:54.818902 6858 tablet_bootstrap.cc:492] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:09:54.819126 6858 ts_tablet_manager.cc:1367] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20240501 20:09:54.820051 6858 raft_consensus.cc:348] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:54.820333 6858 raft_consensus.cc:374] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:54.820489 6858 raft_consensus.cc:729] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:09:54.820777 6858 consensus_queue.cc:260] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:54.820925 6861 ts_tablet_manager.cc:1398] T 7bcb854a8f7545cc86323abd69c26c23 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.010s user 0.002s sys 0.001s
I20240501 20:09:54.822484 6858 ts_tablet_manager.cc:1398] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:54.827874 6775 raft_consensus.cc:482] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:54.828075 6775 raft_consensus.cc:504] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:54.828673 6775 leader_election.cc:288] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:54.829190 4629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bcb854a8f7545cc86323abd69c26c23" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:54.829388 4629 raft_consensus.cc:2454] T 7bcb854a8f7545cc86323abd69c26c23 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:54.829365 4366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bcb854a8f7545cc86323abd69c26c23" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:54.829586 4366 raft_consensus.cc:2454] T 7bcb854a8f7545cc86323abd69c26c23 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:54.829808 4435 leader_election.cc:302] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:54.830088 6775 raft_consensus.cc:2781] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:54.830211 6775 raft_consensus.cc:482] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:54.830300 6775 raft_consensus.cc:3037] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:54.832329 6775 raft_consensus.cc:504] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:54.832883 6775 leader_election.cc:288] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:54.833339 4629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bcb854a8f7545cc86323abd69c26c23" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:54.833341 4366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bcb854a8f7545cc86323abd69c26c23" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:54.833555 4629 raft_consensus.cc:3037] T 7bcb854a8f7545cc86323abd69c26c23 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:54.833626 4366 raft_consensus.cc:3037] T 7bcb854a8f7545cc86323abd69c26c23 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:54.835575 4366 raft_consensus.cc:2454] T 7bcb854a8f7545cc86323abd69c26c23 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:54.835862 4629 raft_consensus.cc:2454] T 7bcb854a8f7545cc86323abd69c26c23 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:54.836141 4434 leader_election.cc:302] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:54.836601 6864 raft_consensus.cc:2781] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:54.836745 6864 raft_consensus.cc:686] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:09:54.837091 6864 consensus_queue.cc:237] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:54.841383 4091 catalog_manager.cc:5484] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:54.990568 6872 raft_consensus.cc:482] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:54.990829 6872 raft_consensus.cc:504] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:54.991497 6872 leader_election.cc:288] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:54.992142 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4f195a6dd14be981395538c2a33ad9" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:09:54.992170 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4f195a6dd14be981395538c2a33ad9" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:54.992374 4365 raft_consensus.cc:2454] T ca4f195a6dd14be981395538c2a33ad9 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:54.992380 4234 raft_consensus.cc:2454] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:54.992794 4434 leader_election.cc:302] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:54.993068 6872 raft_consensus.cc:2781] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:54.993189 6872 raft_consensus.cc:482] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:54.993294 6872 raft_consensus.cc:3037] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:54.995321 6872 raft_consensus.cc:504] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:54.995939 6872 leader_election.cc:288] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Requested vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:09:54.996340 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4f195a6dd14be981395538c2a33ad9" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:54.996479 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4f195a6dd14be981395538c2a33ad9" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:09:54.996556 4233 raft_consensus.cc:3037] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:54.996690 4365 raft_consensus.cc:3037] T ca4f195a6dd14be981395538c2a33ad9 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:54.998492 4233 raft_consensus.cc:2454] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:54.998693 4365 raft_consensus.cc:2454] T ca4f195a6dd14be981395538c2a33ad9 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:54.998896 4434 leader_election.cc:302] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:09:54.999204 6872 raft_consensus.cc:2781] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:54.999424 6872 raft_consensus.cc:686] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:09:54.999706 6872 consensus_queue.cc:237] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:09:55.003283 4091 catalog_manager.cc:5484] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:55.094856 6872 raft_consensus.cc:482] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:55.095036 6872 raft_consensus.cc:504] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:55.095722 6872 leader_election.cc:288] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:09:55.096163 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f99f5d9d4bf44c39188c8b5d5362b98" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:09:55.096280 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f99f5d9d4bf44c39188c8b5d5362b98" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:09:55.096395 4630 raft_consensus.cc:2454] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:55.096524 4233 raft_consensus.cc:2454] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:09:55.096745 4435 leader_election.cc:302] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:55.097031 6872 raft_consensus.cc:2781] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:55.097121 6872 raft_consensus.cc:482] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:55.097225 6872 raft_consensus.cc:3037] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:55.099391 6872 raft_consensus.cc:504] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:55.099917 6872 leader_election.cc:288] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:09:55.100256 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f99f5d9d4bf44c39188c8b5d5362b98" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:09:55.100423 4630 raft_consensus.cc:3037] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:55.100525 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f99f5d9d4bf44c39188c8b5d5362b98" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:09:55.100703 4234 raft_consensus.cc:3037] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:55.102259 4630 raft_consensus.cc:2454] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:55.102723 4234 raft_consensus.cc:2454] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:09:55.102713 4435 leader_election.cc:302] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:09:55.103000 6872 raft_consensus.cc:2781] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:55.103238 6872 raft_consensus.cc:686] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:09:55.103497 6872 consensus_queue.cc:237] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:09:55.107107 4088 catalog_manager.cc:5484] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:55.274494 6864 consensus_queue.cc:1035] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:55.308161 6872 consensus_queue.cc:1035] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:55.517747 6872 consensus_queue.cc:1035] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:55.524113 6775 consensus_queue.cc:1035] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:09:55.579659 6876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:55.580008 6876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:55.588536 6876 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
I20240501 20:09:55.651098 6775 consensus_queue.cc:1035] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:55.663539 6872 consensus_queue.cc:1035] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 271.676 ms
time per row: 0.135838 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:09:57.029556 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.69:46237
--local_ip_for_outbound_sockets=127.29.219.69
--tserver_master_addrs=127.29.219.126:36407
--webserver_port=43169
--webserver_interface=127.29.219.69
--builtin_ntp_servers=127.29.219.84:35419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:09:57.401760 6940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:57.402066 6940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:57.402204 6940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:57.409955 6940 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:57.410092 6940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:57.410260 6940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.69
I20240501 20:09:57.420629 6940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:35419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.69:46237
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.29.219.69
--webserver_port=43169
--tserver_master_addrs=127.29.219.126:36407
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.69
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:57.422128 6940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:57.424098 6940 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:57.439786 6945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:57.440968 6946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:57.442422 6949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:57.442977 6940 server_base.cc:1034] running on GCE node
I20240501 20:09:57.443629 6940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:57.444866 6940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:57.446087 6940 hybrid_clock.cc:648] HybridClock initialized: now 1714594197446017 us; error 61 us; skew 500 ppm
I20240501 20:09:57.446411 6940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:57.454789 6940 webserver.cc:460] Webserver started at http://127.29.219.69:43169/ using document root <none> and password file <none>
I20240501 20:09:57.455773 6940 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:57.455901 6940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:57.462410 6940 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.001s
I20240501 20:09:57.465674 6954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:57.467159 6940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:09:57.467326 6940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-h2cj"
I20240501 20:09:57.468223 6940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:57.507176 6940 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:57.508153 6940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:57.508500 6940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:57.509526 6940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:57.511129 6940 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:57.511210 6940 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:57.511365 6940 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:57.511415 6940 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:57.575222 6940 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.69:46237
I20240501 20:09:57.575278 7066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.69:46237 every 8 connection(s)
I20240501 20:09:57.576848 6940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:09:57.577615 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 6940
I20240501 20:09:57.578424 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.70:36321
--local_ip_for_outbound_sockets=127.29.219.70
--tserver_master_addrs=127.29.219.126:36407
--webserver_port=39459
--webserver_interface=127.29.219.70
--builtin_ntp_servers=127.29.219.84:35419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:57.593704 7067 heartbeater.cc:347] Connected to a master server at 127.29.219.126:36407
I20240501 20:09:57.594058 7067 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:57.594990 7067 heartbeater.cc:511] Master 127.29.219.126:36407 requested a full tablet report, sending...
I20240501 20:09:57.596650 4084 ts_manager.cc:194] Re-registered known tserver with Master: ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:09:57.597671 4084 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.69:60809
W20240501 20:09:57.966277 7071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:57.966578 7071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:57.966732 7071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:57.975538 7071 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:57.975666 7071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:57.975836 7071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.70
I20240501 20:09:57.985348 7071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:35419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.70:36321
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.29.219.70
--webserver_port=39459
--tserver_master_addrs=127.29.219.126:36407
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.70
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:57.986773 7071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:57.988888 7071 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:58.002722 7079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:58.002811 7081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:58.003036 7078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:58.003522 7071 server_base.cc:1034] running on GCE node
I20240501 20:09:58.004179 7071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:58.005522 7071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:58.007138 7071 hybrid_clock.cc:648] HybridClock initialized: now 1714594198006976 us; error 146 us; skew 500 ppm
I20240501 20:09:58.007469 7071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:58.010171 7071 webserver.cc:460] Webserver started at http://127.29.219.70:39459/ using document root <none> and password file <none>
I20240501 20:09:58.011050 7071 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:58.011184 7071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:58.016883 7071 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20240501 20:09:58.019812 7087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:58.021214 7071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 20:09:58.021356 7071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "db0742ee3860468590eefef6ae6074fd"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-h2cj"
I20240501 20:09:58.022205 7071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:58.039878 7071 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:58.040696 7071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:58.040987 7071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:58.041906 7071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:58.043423 7071 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:58.043495 7071 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:58.043587 7071 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:58.043629 7071 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:58.088476 7071 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.70:36321
I20240501 20:09:58.088555 7199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.70:36321 every 8 connection(s)
I20240501 20:09:58.090139 7071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:09:58.096186 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 7071
I20240501 20:09:58.096863 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.71:39447
--local_ip_for_outbound_sockets=127.29.219.71
--tserver_master_addrs=127.29.219.126:36407
--webserver_port=33093
--webserver_interface=127.29.219.71
--builtin_ntp_servers=127.29.219.84:35419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:58.106153 7200 heartbeater.cc:347] Connected to a master server at 127.29.219.126:36407
I20240501 20:09:58.106496 7200 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:58.107386 7200 heartbeater.cc:511] Master 127.29.219.126:36407 requested a full tablet report, sending...
I20240501 20:09:58.108944 4084 ts_manager.cc:194] Re-registered known tserver with Master: db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:09:58.110091 4084 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.70:59861
W20240501 20:09:58.475651 7204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:58.475955 7204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:58.476061 7204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:58.485033 7204 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:58.485160 7204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:58.485277 7204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.71
I20240501 20:09:58.496019 7204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:35419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.71:39447
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.29.219.71
--webserver_port=33093
--tserver_master_addrs=127.29.219.126:36407
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.71
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:58.497459 7204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:58.499682 7204 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:58.513228 7211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:58.513494 7210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:58.514756 7213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:58.515170 7204 server_base.cc:1034] running on GCE node
I20240501 20:09:58.515878 7204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:58.517066 7204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:58.518386 7204 hybrid_clock.cc:648] HybridClock initialized: now 1714594198518168 us; error 203 us; skew 500 ppm
I20240501 20:09:58.518795 7204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:58.521092 7204 webserver.cc:460] Webserver started at http://127.29.219.71:33093/ using document root <none> and password file <none>
I20240501 20:09:58.521854 7204 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:58.521971 7204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:58.528123 7204 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20240501 20:09:58.531157 7219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:58.532730 7204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 20:09:58.532966 7204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "bceeae1c79964acabdae8ff771bf0c97"
format_stamp: "Formatted at 2024-05-01 20:09:23 on dist-test-slave-h2cj"
I20240501 20:09:58.533859 7204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:58.553508 7204 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:58.554498 7204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:58.554817 7204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:58.556242 7204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:58.558035 7204 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:58.558112 7204 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:58.558218 7204 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:58.558279 7204 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:58.592880 7332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.71:39447 every 8 connection(s)
I20240501 20:09:58.592875 7204 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.71:39447
I20240501 20:09:58.594486 7204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 20:09:58.595521 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 7204
I20240501 20:09:58.596254 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.72:39817
--local_ip_for_outbound_sockets=127.29.219.72
--tserver_master_addrs=127.29.219.126:36407
--webserver_port=44747
--webserver_interface=127.29.219.72
--builtin_ntp_servers=127.29.219.84:35419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:09:58.600723 7067 heartbeater.cc:503] Master 127.29.219.126:36407 was elected leader, sending a full tablet report...
I20240501 20:09:58.611582 7333 heartbeater.cc:347] Connected to a master server at 127.29.219.126:36407
I20240501 20:09:58.611979 7333 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:58.613044 7333 heartbeater.cc:511] Master 127.29.219.126:36407 requested a full tablet report, sending...
I20240501 20:09:58.614504 4084 ts_manager.cc:194] Re-registered known tserver with Master: bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:09:58.615597 4084 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.71:43721
W20240501 20:09:58.987803 7337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:58.988153 7337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:09:58.988272 7337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:58.997625 7337 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:09:58.997764 7337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:09:58.997941 7337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.72
I20240501 20:09:59.009711 7337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:35419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.72:39817
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.29.219.72
--webserver_port=44747
--tserver_master_addrs=127.29.219.126:36407
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.72
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:09:59.011111 7337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:09:59.012884 7337 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:09:59.026473 7343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:59.026664 7344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:09:59.027336 7346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:09:59.033003 7337 server_base.cc:1034] running on GCE node
I20240501 20:09:59.033730 7337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:09:59.034781 7337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:09:59.036120 7337 hybrid_clock.cc:648] HybridClock initialized: now 1714594199036041 us; error 73 us; skew 500 ppm
I20240501 20:09:59.036448 7337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:09:59.039505 7337 webserver.cc:460] Webserver started at http://127.29.219.72:44747/ using document root <none> and password file <none>
I20240501 20:09:59.040251 7337 fs_manager.cc:359] Metadata directory not provided
I20240501 20:09:59.040370 7337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:09:59.046097 7337 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20240501 20:09:59.048981 7352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:59.050447 7337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:09:59.050606 7337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "b9957d4086bc4a76b18ed9265a5739d5"
format_stamp: "Formatted at 2024-05-01 20:09:24 on dist-test-slave-h2cj"
I20240501 20:09:59.051448 7337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:09:59.076946 7337 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:09:59.077847 7337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:09:59.078119 7337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:09:59.079139 7337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:09:59.080750 7337 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:09:59.080832 7337 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:59.080960 7337 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:09:59.081017 7337 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:09:59.113082 7200 heartbeater.cc:503] Master 127.29.219.126:36407 was elected leader, sending a full tablet report...
I20240501 20:09:59.148080 7337 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.72:39817
I20240501 20:09:59.148352 7465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.72:39817 every 8 connection(s)
I20240501 20:09:59.150305 7337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 20:09:59.151252 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 7337
I20240501 20:09:59.170975 7466 heartbeater.cc:347] Connected to a master server at 127.29.219.126:36407
I20240501 20:09:59.171334 7466 heartbeater.cc:464] Registering TS with master...
I20240501 20:09:59.171627 4084 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:09:59.172308 7466 heartbeater.cc:511] Master 127.29.219.126:36407 requested a full tablet report, sending...
I20240501 20:09:59.173655 4087 ts_manager.cc:194] Re-registered known tserver with Master: b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:09:59.174723 4087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.72:39819
I20240501 20:09:59.232522 7267 tablet_service.cc:1467] Processing CreateTablet for tablet a63d6f2533d34b7b9ff5ac7c6a5aafd6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=375456659a7d495eaebfc3cf27e3ba75]), partition=
I20240501 20:09:59.233034 7401 tablet_service.cc:1467] Processing CreateTablet for tablet a63d6f2533d34b7b9ff5ac7c6a5aafd6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=375456659a7d495eaebfc3cf27e3ba75]), partition=
I20240501 20:09:59.234300 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a63d6f2533d34b7b9ff5ac7c6a5aafd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:59.234858 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a63d6f2533d34b7b9ff5ac7c6a5aafd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:59.239164 7135 tablet_service.cc:1467] Processing CreateTablet for tablet a63d6f2533d34b7b9ff5ac7c6a5aafd6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=375456659a7d495eaebfc3cf27e3ba75]), partition=
I20240501 20:09:59.240238 7135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a63d6f2533d34b7b9ff5ac7c6a5aafd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:59.265316 7486 tablet_bootstrap.cc:492] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:09:59.272166 7486 tablet_bootstrap.cc:654] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:59.274942 7486 log.cc:824] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:59.280862 7487 tablet_bootstrap.cc:492] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:09:59.281750 7485 tablet_bootstrap.cc:492] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:09:59.286346 7485 tablet_bootstrap.cc:654] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:59.287892 7485 log.cc:824] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:59.288803 7487 tablet_bootstrap.cc:654] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:59.291682 7487 log.cc:824] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:59.297788 7485 tablet_bootstrap.cc:492] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:09:59.298058 7485 ts_tablet_manager.cc:1367] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.005s
I20240501 20:09:59.298290 7486 tablet_bootstrap.cc:492] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:09:59.298569 7486 ts_tablet_manager.cc:1367] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.035s user 0.016s sys 0.000s
I20240501 20:09:59.305133 7485 raft_consensus.cc:348] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.305600 7485 raft_consensus.cc:374] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:59.305706 7485 raft_consensus.cc:729] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:09:59.307060 7485 consensus_queue.cc:260] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.310170 7487 tablet_bootstrap.cc:492] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:09:59.310590 7487 ts_tablet_manager.cc:1367] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.006s
I20240501 20:09:59.315330 7485 ts_tablet_manager.cc:1398] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
I20240501 20:09:59.318096 7486 raft_consensus.cc:348] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.320629 7486 raft_consensus.cc:374] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:59.320753 7486 raft_consensus.cc:729] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:09:59.321465 7486 consensus_queue.cc:260] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.327953 7487 raft_consensus.cc:348] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.328945 7487 raft_consensus.cc:374] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:59.329119 7487 raft_consensus.cc:729] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:09:59.329638 7486 ts_tablet_manager.cc:1398] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.030s user 0.013s sys 0.004s
I20240501 20:09:59.329999 7466 heartbeater.cc:503] Master 127.29.219.126:36407 was elected leader, sending a full tablet report...
I20240501 20:09:59.330058 7487 consensus_queue.cc:260] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.335561 7333 heartbeater.cc:503] Master 127.29.219.126:36407 was elected leader, sending a full tablet report...
I20240501 20:09:59.336350 7487 ts_tablet_manager.cc:1398] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.026s user 0.012s sys 0.004s
W20240501 20:09:59.342923 7201 tablet.cc:2369] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:09:59.347815 7334 tablet.cc:2369] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:09:59.402412 7467 tablet.cc:2369] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:09:59.417806 7492 raft_consensus.cc:482] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:59.418145 7492 raft_consensus.cc:504] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.420008 7492 leader_election.cc:288] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:09:59.427953 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63d6f2533d34b7b9ff5ac7c6a5aafd6" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:09:59.428541 7287 raft_consensus.cc:2454] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:09:59.428805 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63d6f2533d34b7b9ff5ac7c6a5aafd6" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:09:59.429199 7354 leader_election.cc:302] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:09:59.429311 7155 raft_consensus.cc:2454] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:09:59.429641 7492 raft_consensus.cc:2781] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:59.429824 7492 raft_consensus.cc:482] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:59.430294 7492 raft_consensus.cc:3037] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:59.433091 7492 raft_consensus.cc:504] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.433779 7492 leader_election.cc:288] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Requested vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:09:59.434293 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63d6f2533d34b7b9ff5ac7c6a5aafd6" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:09:59.434340 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a63d6f2533d34b7b9ff5ac7c6a5aafd6" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:09:59.434532 7155 raft_consensus.cc:3037] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:59.434515 7287 raft_consensus.cc:3037] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:59.436869 7155 raft_consensus.cc:2454] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:09:59.437454 7353 leader_election.cc:302] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:09:59.437716 7492 raft_consensus.cc:2781] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:59.438284 7492 raft_consensus.cc:686] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Becoming Leader. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Running, Role: LEADER
I20240501 20:09:59.438365 7287 raft_consensus.cc:2454] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:09:59.439245 7492 consensus_queue.cc:237] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.444375 4084 catalog_manager.cc:5484] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72). New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } }
I20240501 20:09:59.559355 4084 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:09:59.569643 7155 raft_consensus.cc:1261] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:59.570111 7287 raft_consensus.cc:1261] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:09:59.571007 7495 consensus_queue.cc:1035] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:59.571693 7495 consensus_queue.cc:1035] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:09:59.587704 4084 catalog_manager.cc:5576] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321) does not have the latest schema for tablet a63d6f2533d34b7b9ff5ac7c6a5aafd6 (table rebalancer_dynamic_table [id=375456659a7d495eaebfc3cf27e3ba75]). Expected version 1 got 0
I20240501 20:09:59.590914 7499 mvcc.cc:205] Tried to move back new op lower bound from 7022977841425199104 to 7022977840907554816. Current Snapshot: MvccSnapshot[applied={T|T < 7022977841425199104}]
I20240501 20:09:59.594492 7502 tablet.cc:1722] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:59.597492 7502 tablet.cc:1620] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:09:59.598577 4087 catalog_manager.cc:5576] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447) does not have the latest schema for tablet a63d6f2533d34b7b9ff5ac7c6a5aafd6 (table rebalancer_dynamic_table [id=375456659a7d495eaebfc3cf27e3ba75]). Expected version 1 got 0
I20240501 20:09:59.604207 4084 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=375456659a7d495eaebfc3cf27e3ba75] alter complete (version 1)
I20240501 20:09:59.606774 7497 mvcc.cc:205] Tried to move back new op lower bound from 7022977841425199104 to 7022977840907554816. Current Snapshot: MvccSnapshot[applied={T|T < 7022977841425199104}]
I20240501 20:09:59.621104 7504 tablet.cc:1722] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:59.624660 7503 tablet.cc:1722] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:09:59.626027 4084 catalog_manager.cc:5576] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447) does not have the latest schema for tablet a63d6f2533d34b7b9ff5ac7c6a5aafd6 (table rebalancer_dynamic_table [id=375456659a7d495eaebfc3cf27e3ba75]). Expected version 1 got 0
I20240501 20:09:59.628252 4084 catalog_manager.cc:5576] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321) does not have the latest schema for tablet a63d6f2533d34b7b9ff5ac7c6a5aafd6 (table rebalancer_dynamic_table [id=375456659a7d495eaebfc3cf27e3ba75]). Expected version 1 got 0
I20240501 20:09:59.638072 7503 tablet.cc:1620] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:09:59.639999 7504 tablet.cc:1620] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:09:59.662376 4084 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:09:59.669723 4088 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240501 20:09:59.672066 4088 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240501 20:09:59.675675 7267 tablet_service.cc:1467] Processing CreateTablet for tablet d1291a2b3dcc4c55a26df6a5cc390050 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=40f3268f46a14344bdf5de27989a6f1a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:59.675621 7401 tablet_service.cc:1467] Processing CreateTablet for tablet d1291a2b3dcc4c55a26df6a5cc390050 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=40f3268f46a14344bdf5de27989a6f1a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:59.676321 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1291a2b3dcc4c55a26df6a5cc390050. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:59.676357 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1291a2b3dcc4c55a26df6a5cc390050. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:59.679723 7133 tablet_service.cc:1467] Processing CreateTablet for tablet 5215e3f2040747f29aa1c9181a628096 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=40f3268f46a14344bdf5de27989a6f1a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:59.680541 7133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5215e3f2040747f29aa1c9181a628096. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:59.681720 7134 tablet_service.cc:1467] Processing CreateTablet for tablet bb85d7825c564339bd554deeded45cec (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=40f3268f46a14344bdf5de27989a6f1a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:59.682353 7134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb85d7825c564339bd554deeded45cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:59.685379 7487 tablet_bootstrap.cc:492] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:09:59.687322 7135 tablet_service.cc:1467] Processing CreateTablet for tablet d1291a2b3dcc4c55a26df6a5cc390050 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=40f3268f46a14344bdf5de27989a6f1a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:59.687559 7487 tablet_bootstrap.cc:654] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:59.687723 7135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1291a2b3dcc4c55a26df6a5cc390050. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:59.692898 7266 tablet_service.cc:1467] Processing CreateTablet for tablet 5215e3f2040747f29aa1c9181a628096 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=40f3268f46a14344bdf5de27989a6f1a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:59.693536 7266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5215e3f2040747f29aa1c9181a628096. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:59.693707 7267 tablet_service.cc:1467] Processing CreateTablet for tablet bb85d7825c564339bd554deeded45cec (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=40f3268f46a14344bdf5de27989a6f1a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:59.694200 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb85d7825c564339bd554deeded45cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:59.714108 4084 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:09:59.717029 4088 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet c7e9b168477a4dce9c70ed20468711d9
I20240501 20:09:59.718322 4346 tablet_service.cc:1514] Processing DeleteTablet for tablet c7e9b168477a4dce9c70ed20468711d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:59 UTC) from {username='slave'} at 127.0.0.1:60928
I20240501 20:09:59.720103 7485 tablet_bootstrap.cc:492] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:09:59.721909 7485 tablet_bootstrap.cc:654] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:59.728068 7502 tablet.cc:1722] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:59.728641 7502 tablet.cc:1620] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:09:59.730057 7504 tablet.cc:1722] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:59.730583 7504 tablet.cc:1620] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:09:59.733923 7503 tablet.cc:1722] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:09:59.734493 7503 tablet.cc:1620] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:09:59.742808 4084 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=375456659a7d495eaebfc3cf27e3ba75] alter complete (version 2)
I20240501 20:09:59.747881 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet c7e9b168477a4dce9c70ed20468711d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:59 UTC) from {username='slave'} at 127.0.0.1:55490
I20240501 20:09:59.748700 7486 tablet_bootstrap.cc:492] T d1291a2b3dcc4c55a26df6a5cc390050 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:09:59.750551 7486 tablet_bootstrap.cc:654] T d1291a2b3dcc4c55a26df6a5cc390050 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:59.757858 7485 tablet_bootstrap.cc:492] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:09:59.758389 7485 ts_tablet_manager.cc:1367] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.038s user 0.010s sys 0.005s
I20240501 20:09:59.759646 7485 raft_consensus.cc:348] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:09:59.761144 7485 raft_consensus.cc:374] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:59.761432 7485 raft_consensus.cc:729] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:09:59.762084 7514 tablet_replica.cc:330] T c7e9b168477a4dce9c70ed20468711d9 P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:09:59.762472 7485 consensus_queue.cc:260] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:09:59.765456 7485 ts_tablet_manager.cc:1398] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 20:09:59.766032 7514 raft_consensus.cc:2229] T c7e9b168477a4dce9c70ed20468711d9 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:59.768651 7485 tablet_bootstrap.cc:492] T d1291a2b3dcc4c55a26df6a5cc390050 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:09:59.769125 7514 raft_consensus.cc:2258] T c7e9b168477a4dce9c70ed20468711d9 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:59.770175 4611 tablet_service.cc:1514] Processing DeleteTablet for tablet c7e9b168477a4dce9c70ed20468711d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:59 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:09:59.770987 7485 tablet_bootstrap.cc:654] T d1291a2b3dcc4c55a26df6a5cc390050 P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:59.773921 7514 ts_tablet_manager.cc:1875] T c7e9b168477a4dce9c70ed20468711d9 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:59.774430 4088 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet bfd49691236341dea4af4ead4450286e
I20240501 20:09:59.775496 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet bfd49691236341dea4af4ead4450286e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:59 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:09:59.775789 4088 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 8ddee2fd0dfd4e3295cbfad7edd5add6
I20240501 20:09:59.776578 4213 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ddee2fd0dfd4e3295cbfad7edd5add6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:59 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:09:59.776623 4348 tablet_service.cc:1514] Processing DeleteTablet for tablet bfd49691236341dea4af4ead4450286e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:59 UTC) from {username='slave'} at 127.0.0.1:60928
I20240501 20:09:59.776883 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ddee2fd0dfd4e3295cbfad7edd5add6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:59 UTC) from {username='slave'} at 127.0.0.1:60928
I20240501 20:09:59.777935 4609 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ddee2fd0dfd4e3295cbfad7edd5add6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:59 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:09:59.778270 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet bfd49691236341dea4af4ead4450286e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:59 UTC) from {username='slave'} at 127.0.0.1:55490
I20240501 20:09:59.783759 7001 tablet_service.cc:1467] Processing CreateTablet for tablet 5215e3f2040747f29aa1c9181a628096 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=40f3268f46a14344bdf5de27989a6f1a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:59.785538 7001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5215e3f2040747f29aa1c9181a628096. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:59.786190 7485 tablet_bootstrap.cc:492] T d1291a2b3dcc4c55a26df6a5cc390050 P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:09:59.786356 7485 ts_tablet_manager.cc:1367] T d1291a2b3dcc4c55a26df6a5cc390050 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240501 20:09:59.786569 7487 tablet_bootstrap.cc:492] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:09:59.786748 7487 ts_tablet_manager.cc:1367] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.101s user 0.009s sys 0.000s
I20240501 20:09:59.787169 7485 raft_consensus.cc:348] T d1291a2b3dcc4c55a26df6a5cc390050 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.787397 7485 raft_consensus.cc:374] T d1291a2b3dcc4c55a26df6a5cc390050 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:59.787460 7485 raft_consensus.cc:729] T d1291a2b3dcc4c55a26df6a5cc390050 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:09:59.787710 7487 raft_consensus.cc:348] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.787755 7485 consensus_queue.cc:260] T d1291a2b3dcc4c55a26df6a5cc390050 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.787950 7487 raft_consensus.cc:374] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:59.788050 7487 raft_consensus.cc:729] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:09:59.788265 7487 consensus_queue.cc:260] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.788774 7487 ts_tablet_manager.cc:1398] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:09:59.789076 7487 tablet_bootstrap.cc:492] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:09:59.789317 7485 ts_tablet_manager.cc:1398] T d1291a2b3dcc4c55a26df6a5cc390050 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:09:59.789598 7485 tablet_bootstrap.cc:492] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:09:59.790316 7512 tablet_replica.cc:330] T c7e9b168477a4dce9c70ed20468711d9 P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:09:59.790880 7512 raft_consensus.cc:2229] T c7e9b168477a4dce9c70ed20468711d9 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:59.791149 7485 tablet_bootstrap.cc:654] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:59.792598 7486 tablet_bootstrap.cc:492] T d1291a2b3dcc4c55a26df6a5cc390050 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:09:59.793206 7486 ts_tablet_manager.cc:1367] T d1291a2b3dcc4c55a26df6a5cc390050 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.005s
I20240501 20:09:59.793704 7487 tablet_bootstrap.cc:654] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:59.794551 7486 raft_consensus.cc:348] T d1291a2b3dcc4c55a26df6a5cc390050 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.792102 7002 tablet_service.cc:1467] Processing CreateTablet for tablet bb85d7825c564339bd554deeded45cec (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=40f3268f46a14344bdf5de27989a6f1a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:09:59.795313 7517 tablet_replica.cc:330] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:09:59.795913 7517 raft_consensus.cc:2229] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:59.796348 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb85d7825c564339bd554deeded45cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:09:59.796628 7517 raft_consensus.cc:2258] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:59.797024 7519 tablet_replica.cc:330] T bfd49691236341dea4af4ead4450286e P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:09:59.797873 7519 raft_consensus.cc:2229] T bfd49691236341dea4af4ead4450286e P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:59.798198 7519 raft_consensus.cc:2258] T bfd49691236341dea4af4ead4450286e P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:59.798274 7517 ts_tablet_manager.cc:1875] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:59.798708 7486 raft_consensus.cc:374] T d1291a2b3dcc4c55a26df6a5cc390050 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:59.798830 7486 raft_consensus.cc:729] T d1291a2b3dcc4c55a26df6a5cc390050 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:09:59.799096 7486 consensus_queue.cc:260] T d1291a2b3dcc4c55a26df6a5cc390050 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.799283 7514 ts_tablet_manager.cc:1888] T c7e9b168477a4dce9c70ed20468711d9 P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:59.799448 7514 log.cc:1196] T c7e9b168477a4dce9c70ed20468711d9 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/c7e9b168477a4dce9c70ed20468711d9
I20240501 20:09:59.799644 7486 ts_tablet_manager.cc:1398] T d1291a2b3dcc4c55a26df6a5cc390050 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 20:09:59.799659 7519 ts_tablet_manager.cc:1875] T bfd49691236341dea4af4ead4450286e P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:59.800006 7514 ts_tablet_manager.cc:1909] T c7e9b168477a4dce9c70ed20468711d9 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting consensus metadata
I20240501 20:09:59.800593 7512 raft_consensus.cc:2258] T c7e9b168477a4dce9c70ed20468711d9 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:59.801748 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet c7e9b168477a4dce9c70ed20468711d9 (table rebalance_test_table_0 [id=c8faa922e6eb4703821285891838ebab]) successfully deleted
I20240501 20:09:59.802078 7512 ts_tablet_manager.cc:1875] T c7e9b168477a4dce9c70ed20468711d9 P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:59.807653 7485 tablet_bootstrap.cc:492] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:09:59.807819 7485 ts_tablet_manager.cc:1367] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240501 20:09:59.808601 7485 raft_consensus.cc:348] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.808840 7485 raft_consensus.cc:374] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:59.808928 7485 raft_consensus.cc:729] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:09:59.809190 7485 consensus_queue.cc:260] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.812243 7485 ts_tablet_manager.cc:1398] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240501 20:09:59.815397 7514 tablet_replica.cc:330] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:09:59.815717 7514 raft_consensus.cc:2229] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:59.815811 7487 tablet_bootstrap.cc:492] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:09:59.816002 7487 ts_tablet_manager.cc:1367] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.002s
I20240501 20:09:59.816514 7514 raft_consensus.cc:2258] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:59.816780 7487 raft_consensus.cc:348] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.817030 7487 raft_consensus.cc:374] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:59.817636 7487 raft_consensus.cc:729] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:09:59.817693 7519 ts_tablet_manager.cc:1888] T bfd49691236341dea4af4ead4450286e P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:59.817759 7517 ts_tablet_manager.cc:1888] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:59.817916 7517 log.cc:1196] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/c7e9b168477a4dce9c70ed20468711d9
I20240501 20:09:59.817948 7519 log.cc:1196] T bfd49691236341dea4af4ead4450286e P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/bfd49691236341dea4af4ead4450286e
I20240501 20:09:59.818071 7487 consensus_queue.cc:260] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.818346 7512 ts_tablet_manager.cc:1888] T c7e9b168477a4dce9c70ed20468711d9 P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:59.818480 7519 ts_tablet_manager.cc:1909] T bfd49691236341dea4af4ead4450286e P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:09:59.818508 7514 ts_tablet_manager.cc:1875] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:59.818558 7512 log.cc:1196] T c7e9b168477a4dce9c70ed20468711d9 P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/c7e9b168477a4dce9c70ed20468711d9
I20240501 20:09:59.819096 7512 ts_tablet_manager.cc:1909] T c7e9b168477a4dce9c70ed20468711d9 P 10b47782558c41ee91f5e164a372049a: Deleting consensus metadata
I20240501 20:09:59.820631 7512 tablet_replica.cc:330] T bfd49691236341dea4af4ead4450286e P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:09:59.820592 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet bfd49691236341dea4af4ead4450286e (table rebalance_test_table_0 [id=c8faa922e6eb4703821285891838ebab]) successfully deleted
I20240501 20:09:59.820749 7487 ts_tablet_manager.cc:1398] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240501 20:09:59.821203 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet c7e9b168477a4dce9c70ed20468711d9 (table rebalance_test_table_0 [id=c8faa922e6eb4703821285891838ebab]) successfully deleted
I20240501 20:09:59.821416 7517 ts_tablet_manager.cc:1909] T c7e9b168477a4dce9c70ed20468711d9 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:09:59.822576 7487 tablet_bootstrap.cc:492] T bb85d7825c564339bd554deeded45cec P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:09:59.822939 7512 raft_consensus.cc:2229] T bfd49691236341dea4af4ead4450286e P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:59.823267 7512 raft_consensus.cc:2258] T bfd49691236341dea4af4ead4450286e P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:59.823983 7519 tablet_replica.cc:330] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:09:59.824213 7487 tablet_bootstrap.cc:654] T bb85d7825c564339bd554deeded45cec P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:59.824218 7519 raft_consensus.cc:2229] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:09:59.824651 7519 raft_consensus.cc:2258] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:59.826169 7519 ts_tablet_manager.cc:1875] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:59.827237 7512 ts_tablet_manager.cc:1875] T bfd49691236341dea4af4ead4450286e P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:59.827723 7517 tablet_replica.cc:330] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:09:59.827924 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet c7e9b168477a4dce9c70ed20468711d9 (table rebalance_test_table_0 [id=c8faa922e6eb4703821285891838ebab]) successfully deleted
I20240501 20:09:59.828833 7517 raft_consensus.cc:2229] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:59.829250 7517 raft_consensus.cc:2258] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:59.830725 7517 ts_tablet_manager.cc:1875] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:59.831223 7493 raft_consensus.cc:482] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:59.831409 7493 raft_consensus.cc:504] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.833175 7514 ts_tablet_manager.cc:1888] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:59.833388 7514 log.cc:1196] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/bfd49691236341dea4af4ead4450286e
I20240501 20:09:59.833621 7493 leader_election.cc:288] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:09:59.838693 7514 ts_tablet_manager.cc:1909] T bfd49691236341dea4af4ead4450286e P 3e974c03d3c546f9ab6c91e762a5426b: Deleting consensus metadata
I20240501 20:09:59.839830 7519 ts_tablet_manager.cc:1888] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:59.839977 7519 log.cc:1196] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/8ddee2fd0dfd4e3295cbfad7edd5add6
I20240501 20:09:59.840433 7519 ts_tablet_manager.cc:1909] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:09:59.841529 7517 ts_tablet_manager.cc:1888] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:59.842092 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet bfd49691236341dea4af4ead4450286e (table rebalance_test_table_0 [id=c8faa922e6eb4703821285891838ebab]) successfully deleted
I20240501 20:09:59.843648 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet 8ddee2fd0dfd4e3295cbfad7edd5add6 (table rebalance_test_table_0 [id=c8faa922e6eb4703821285891838ebab]) successfully deleted
I20240501 20:09:59.844318 7512 ts_tablet_manager.cc:1888] T bfd49691236341dea4af4ead4450286e P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:59.844516 7512 log.cc:1196] T bfd49691236341dea4af4ead4450286e P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/bfd49691236341dea4af4ead4450286e
I20240501 20:09:59.844935 7512 ts_tablet_manager.cc:1909] T bfd49691236341dea4af4ead4450286e P 10b47782558c41ee91f5e164a372049a: Deleting consensus metadata
I20240501 20:09:59.845683 7512 tablet_replica.cc:330] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:09:59.845773 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet bfd49691236341dea4af4ead4450286e (table rebalance_test_table_0 [id=c8faa922e6eb4703821285891838ebab]) successfully deleted
I20240501 20:09:59.845866 7517 log.cc:1196] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/8ddee2fd0dfd4e3295cbfad7edd5add6
I20240501 20:09:59.846302 7517 ts_tablet_manager.cc:1909] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:09:59.847364 7512 raft_consensus.cc:2229] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:09:59.847679 7512 raft_consensus.cc:2258] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:09:59.847642 4087 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:09:59.848482 7512 ts_tablet_manager.cc:1875] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:09:59.849048 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet 8ddee2fd0dfd4e3295cbfad7edd5add6 (table rebalance_test_table_0 [id=c8faa922e6eb4703821285891838ebab]) successfully deleted
I20240501 20:09:59.854423 7512 ts_tablet_manager.cc:1888] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:09:59.854699 7512 log.cc:1196] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/8ddee2fd0dfd4e3295cbfad7edd5add6
I20240501 20:09:59.855265 7512 ts_tablet_manager.cc:1909] T 8ddee2fd0dfd4e3295cbfad7edd5add6 P 10b47782558c41ee91f5e164a372049a: Deleting consensus metadata
I20240501 20:09:59.858120 7502 tablet.cc:1722] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:59.858393 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet 8ddee2fd0dfd4e3295cbfad7edd5add6 (table rebalance_test_table_0 [id=c8faa922e6eb4703821285891838ebab]) successfully deleted
I20240501 20:09:59.861138 7504 tablet.cc:1722] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:59.861260 7503 tablet.cc:1722] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:09:59.869108 7526 tablet_bootstrap.cc:492] T bb85d7825c564339bd554deeded45cec P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:09:59.869437 7491 raft_consensus.cc:482] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:59.869654 7491 raft_consensus.cc:504] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:09:59.871698 4087 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table_new_name [id=375456659a7d495eaebfc3cf27e3ba75] alter complete (version 3)
I20240501 20:09:59.871867 7491 leader_election.cc:288] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:09:59.872834 7491 raft_consensus.cc:482] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:59.873034 7491 raft_consensus.cc:504] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.873560 7526 tablet_bootstrap.cc:654] T bb85d7825c564339bd554deeded45cec P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:59.873659 7491 leader_election.cc:288] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:09:59.875174 7526 log.cc:824] T bb85d7825c564339bd554deeded45cec P ef0a4ada548446b3a531cb39a3c4e38d: Log is configured to *not* fsync() on all Append() calls
I20240501 20:09:59.881383 7526 tablet_bootstrap.cc:492] T bb85d7825c564339bd554deeded45cec P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:09:59.881942 7526 ts_tablet_manager.cc:1367] T bb85d7825c564339bd554deeded45cec P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s
I20240501 20:09:59.904667 7493 raft_consensus.cc:482] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:09:59.905174 7493 raft_consensus.cc:504] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.906543 7493 leader_election.cc:288] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:09:59.916437 7526 raft_consensus.cc:348] T bb85d7825c564339bd554deeded45cec P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:09:59.917140 7526 raft_consensus.cc:374] T bb85d7825c564339bd554deeded45cec P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:59.917333 7526 raft_consensus.cc:729] T bb85d7825c564339bd554deeded45cec P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:09:59.918507 7526 consensus_queue.cc:260] T bb85d7825c564339bd554deeded45cec P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:09:59.933195 7526 ts_tablet_manager.cc:1398] T bb85d7825c564339bd554deeded45cec P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.049s user 0.014s sys 0.011s
I20240501 20:09:59.933719 7526 tablet_bootstrap.cc:492] T 5215e3f2040747f29aa1c9181a628096 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:09:59.935909 7526 tablet_bootstrap.cc:654] T 5215e3f2040747f29aa1c9181a628096 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:09:59.946403 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb85d7825c564339bd554deeded45cec" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:09:59.947161 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5215e3f2040747f29aa1c9181a628096" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:09:59.947389 7287 raft_consensus.cc:2454] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
W20240501 20:09:59.948165 7089 leader_election.cc:341] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:59.948516 7089 leader_election.cc:302] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:09:59.949014 7491 raft_consensus.cc:2781] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:59.949115 7491 raft_consensus.cc:482] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:59.949221 7491 raft_consensus.cc:3037] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:59.952126 7491 raft_consensus.cc:504] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.953068 7491 leader_election.cc:288] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:09:59.953704 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5215e3f2040747f29aa1c9181a628096" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:09:59.953941 7287 raft_consensus.cc:3037] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:59.955804 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5215e3f2040747f29aa1c9181a628096" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
W20240501 20:09:59.957600 7222 leader_election.cc:341] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:09:59.957878 7287 raft_consensus.cc:2454] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:09:59.959074 7089 leader_election.cc:302] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:09:59.959388 7491 raft_consensus.cc:2781] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:59.959954 7154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5215e3f2040747f29aa1c9181a628096" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:09:59.962715 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1291a2b3dcc4c55a26df6a5cc390050" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:09:59.963013 7155 raft_consensus.cc:2454] T d1291a2b3dcc4c55a26df6a5cc390050 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:09:59.963608 7220 leader_election.cc:302] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:09:59.964046 7493 raft_consensus.cc:2781] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:59.964141 7493 raft_consensus.cc:482] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:59.964236 7493 raft_consensus.cc:3037] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:59.965181 7526 tablet_bootstrap.cc:492] T 5215e3f2040747f29aa1c9181a628096 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:09:59.965376 7526 ts_tablet_manager.cc:1367] T 5215e3f2040747f29aa1c9181a628096 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.007s
I20240501 20:09:59.967411 7493 raft_consensus.cc:504] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.969297 7487 tablet_bootstrap.cc:492] T bb85d7825c564339bd554deeded45cec P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:09:59.969308 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1291a2b3dcc4c55a26df6a5cc390050" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:09:59.969447 7487 ts_tablet_manager.cc:1367] T bb85d7825c564339bd554deeded45cec P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.147s user 0.021s sys 0.010s
I20240501 20:09:59.969485 7155 raft_consensus.cc:3037] T d1291a2b3dcc4c55a26df6a5cc390050 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:59.969497 7421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1291a2b3dcc4c55a26df6a5cc390050" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:09:59.969952 7421 raft_consensus.cc:2454] T d1291a2b3dcc4c55a26df6a5cc390050 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:09:59.970242 7487 raft_consensus.cc:348] T bb85d7825c564339bd554deeded45cec P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:09:59.970425 7487 raft_consensus.cc:374] T bb85d7825c564339bd554deeded45cec P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:59.970503 7487 raft_consensus.cc:729] T bb85d7825c564339bd554deeded45cec P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:09:59.970395 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1291a2b3dcc4c55a26df6a5cc390050" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:09:59.970674 7420 raft_consensus.cc:3037] T d1291a2b3dcc4c55a26df6a5cc390050 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:59.970752 7487 consensus_queue.cc:260] T bb85d7825c564339bd554deeded45cec P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:09:59.972014 7526 raft_consensus.cc:348] T 5215e3f2040747f29aa1c9181a628096 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.972250 7526 raft_consensus.cc:374] T 5215e3f2040747f29aa1c9181a628096 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:09:59.972330 7526 raft_consensus.cc:729] T 5215e3f2040747f29aa1c9181a628096 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:09:59.972528 7526 consensus_queue.cc:260] T 5215e3f2040747f29aa1c9181a628096 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.972713 7155 raft_consensus.cc:2454] T d1291a2b3dcc4c55a26df6a5cc390050 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:09:59.973524 7420 raft_consensus.cc:2454] T d1291a2b3dcc4c55a26df6a5cc390050 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:09:59.973875 7526 ts_tablet_manager.cc:1398] T 5215e3f2040747f29aa1c9181a628096 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20240501 20:09:59.974292 7021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5215e3f2040747f29aa1c9181a628096" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:09:59.974843 7021 raft_consensus.cc:2454] T 5215e3f2040747f29aa1c9181a628096 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:09:59.975355 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5215e3f2040747f29aa1c9181a628096" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:09:59.975652 7020 raft_consensus.cc:3037] T 5215e3f2040747f29aa1c9181a628096 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:59.975811 7223 leader_election.cc:302] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:09:59.976167 7491 raft_consensus.cc:686] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Becoming Leader. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Running, Role: LEADER
I20240501 20:09:59.976701 7491 consensus_queue.cc:237] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:09:59.977126 7545 raft_consensus.cc:2781] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:09:59.977272 7545 raft_consensus.cc:686] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Becoming Leader. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Running, Role: LEADER
I20240501 20:09:59.978886 4087 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:59.979187 4087 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:09:59.985082 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb85d7825c564339bd554deeded45cec" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:09:59.985406 7022 raft_consensus.cc:2454] T bb85d7825c564339bd554deeded45cec P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:09:59.986371 7090 leader_election.cc:302] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db0742ee3860468590eefef6ae6074fd, ef0a4ada548446b3a531cb39a3c4e38d; no voters: bceeae1c79964acabdae8ff771bf0c97
I20240501 20:09:59.987185 7543 raft_consensus.cc:2781] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:09:59.987320 7543 raft_consensus.cc:482] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:09:59.987449 7543 raft_consensus.cc:3037] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:09:59.989737 4087 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet a63d6f2533d34b7b9ff5ac7c6a5aafd6
I20240501 20:09:59.993525 7401 tablet_service.cc:1514] Processing DeleteTablet for tablet a63d6f2533d34b7b9ff5ac7c6a5aafd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:59 UTC) from {username='slave'} at 127.0.0.1:40182
I20240501 20:09:59.997051 7020 raft_consensus.cc:2454] T 5215e3f2040747f29aa1c9181a628096 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:09:59.998315 7543 raft_consensus.cc:504] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:00.002053 7133 tablet_service.cc:1514] Processing DeleteTablet for tablet a63d6f2533d34b7b9ff5ac7c6a5aafd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:59 UTC) from {username='slave'} at 127.0.0.1:52036
I20240501 20:10:00.012737 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb85d7825c564339bd554deeded45cec" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:00.013212 7020 raft_consensus.cc:3037] T bb85d7825c564339bd554deeded45cec P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:00.017740 7543 leader_election.cc:288] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Requested vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:00.021905 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb85d7825c564339bd554deeded45cec" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:00.023494 7493 leader_election.cc:288] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Requested vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:00.023963 7020 raft_consensus.cc:2454] T bb85d7825c564339bd554deeded45cec P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:00.023896 7545 consensus_queue.cc:237] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:00.025050 7487 ts_tablet_manager.cc:1398] T bb85d7825c564339bd554deeded45cec P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.052s user 0.013s sys 0.038s
I20240501 20:10:00.025099 7267 tablet_service.cc:1514] Processing DeleteTablet for tablet a63d6f2533d34b7b9ff5ac7c6a5aafd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:09:59 UTC) from {username='slave'} at 127.0.0.1:54166
I20240501 20:10:00.025648 7287 raft_consensus.cc:3037] T bb85d7825c564339bd554deeded45cec P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:00.025131 7547 tablet_replica.cc:330] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5: stopping tablet replica
I20240501 20:10:00.027904 7547 raft_consensus.cc:2229] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:00.028508 7287 raft_consensus.cc:2454] T bb85d7825c564339bd554deeded45cec P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:00.028627 7547 raft_consensus.cc:2258] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:00.029237 7089 leader_election.cc:302] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:00.034780 7543 raft_consensus.cc:2781] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:00.038491 7547 ts_tablet_manager.cc:1875] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:00.039435 4088 catalog_manager.cc:5484] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd reported cstate change: term changed from 0 to 1, leader changed from <none> to db0742ee3860468590eefef6ae6074fd (127.29.219.70). New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:00.043694 7220 leader_election.cc:302] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97; no voters: db0742ee3860468590eefef6ae6074fd, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:00.045670 7493 raft_consensus.cc:2726] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:00.055894 7549 tablet_replica.cc:330] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:00.056561 7549 raft_consensus.cc:2229] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:00.056942 7549 raft_consensus.cc:2258] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:00.059420 7543 raft_consensus.cc:686] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Becoming Leader. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Running, Role: LEADER
I20240501 20:10:00.059947 7543 consensus_queue.cc:237] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:00.061028 7549 ts_tablet_manager.cc:1875] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:00.061249 4088 catalog_manager.cc:5484] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71). New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:00.062824 7547 ts_tablet_manager.cc:1888] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:00.062992 7547 log.cc:1196] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/wals/a63d6f2533d34b7b9ff5ac7c6a5aafd6
I20240501 20:10:00.063555 7547 ts_tablet_manager.cc:1909] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P b9957d4086bc4a76b18ed9265a5739d5: Deleting consensus metadata
I20240501 20:10:00.065333 4079 catalog_manager.cc:4830] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): tablet a63d6f2533d34b7b9ff5ac7c6a5aafd6 (table rebalancer_dynamic_table_new_name [id=375456659a7d495eaebfc3cf27e3ba75]) successfully deleted
I20240501 20:10:00.066722 7548 tablet_replica.cc:330] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd: stopping tablet replica
I20240501 20:10:00.067203 7548 raft_consensus.cc:2229] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:00.067533 7548 raft_consensus.cc:2258] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:00.069454 7548 ts_tablet_manager.cc:1875] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:00.071256 7549 ts_tablet_manager.cc:1888] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:00.071460 7549 log.cc:1196] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/a63d6f2533d34b7b9ff5ac7c6a5aafd6
I20240501 20:10:00.072119 7549 ts_tablet_manager.cc:1909] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P bceeae1c79964acabdae8ff771bf0c97: Deleting consensus metadata
I20240501 20:10:00.074779 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet a63d6f2533d34b7b9ff5ac7c6a5aafd6 (table rebalancer_dynamic_table_new_name [id=375456659a7d495eaebfc3cf27e3ba75]) successfully deleted
I20240501 20:10:00.076965 7548 ts_tablet_manager.cc:1888] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:00.077180 7548 log.cc:1196] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/wals/a63d6f2533d34b7b9ff5ac7c6a5aafd6
I20240501 20:10:00.077728 7548 ts_tablet_manager.cc:1909] T a63d6f2533d34b7b9ff5ac7c6a5aafd6 P db0742ee3860468590eefef6ae6074fd: Deleting consensus metadata
I20240501 20:10:00.079514 4076 catalog_manager.cc:4830] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): tablet a63d6f2533d34b7b9ff5ac7c6a5aafd6 (table rebalancer_dynamic_table_new_name [id=375456659a7d495eaebfc3cf27e3ba75]) successfully deleted
W20240501 20:10:00.080188 7068 tablet.cc:2369] T 5215e3f2040747f29aa1c9181a628096 P ef0a4ada548446b3a531cb39a3c4e38d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:10:00.082942 4087 catalog_manager.cc:5484] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd reported cstate change: term changed from 0 to 1, leader changed from <none> to db0742ee3860468590eefef6ae6074fd (127.29.219.70). New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:00.096781 4087 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:00.110090 4215 tablet_service.cc:1467] Processing CreateTablet for tablet a778d21259eb42e68c8b0fd534facc5b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c133c6c195dc43b6891545bfdf601e5d]), partition=
I20240501 20:10:00.110458 7133 tablet_service.cc:1467] Processing CreateTablet for tablet a778d21259eb42e68c8b0fd534facc5b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c133c6c195dc43b6891545bfdf601e5d]), partition=
I20240501 20:10:00.110517 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a778d21259eb42e68c8b0fd534facc5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:00.110921 7133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a778d21259eb42e68c8b0fd534facc5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:00.114988 7401 tablet_service.cc:1467] Processing CreateTablet for tablet a778d21259eb42e68c8b0fd534facc5b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c133c6c195dc43b6891545bfdf601e5d]), partition=
I20240501 20:10:00.115365 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a778d21259eb42e68c8b0fd534facc5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:00.117851 7485 tablet_bootstrap.cc:492] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:00.119500 7485 tablet_bootstrap.cc:654] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:00.121292 7486 tablet_bootstrap.cc:492] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:00.122762 7486 tablet_bootstrap.cc:654] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:00.124151 7485 tablet_bootstrap.cc:492] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:00.124325 7485 ts_tablet_manager.cc:1367] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240501 20:10:00.125093 7485 raft_consensus.cc:348] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:00.125336 7485 raft_consensus.cc:374] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:00.125425 7485 raft_consensus.cc:729] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:00.126569 7485 consensus_queue.cc:260] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:00.129881 7485 ts_tablet_manager.cc:1398] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240501 20:10:00.133365 7486 tablet_bootstrap.cc:492] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:00.133574 7486 ts_tablet_manager.cc:1367] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20240501 20:10:00.134335 7486 raft_consensus.cc:348] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:00.134567 7486 raft_consensus.cc:374] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:00.134677 7486 raft_consensus.cc:729] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:00.134941 7486 consensus_queue.cc:260] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:00.138345 7486 ts_tablet_manager.cc:1398] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:10:00.141949 7560 tablet_bootstrap.cc:492] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:00.143900 7560 tablet_bootstrap.cc:654] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:00.145965 7560 tablet_bootstrap.cc:492] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:00.146136 7560 ts_tablet_manager.cc:1367] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:10:00.146725 7560 raft_consensus.cc:348] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:00.146890 7560 raft_consensus.cc:374] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:00.146971 7560 raft_consensus.cc:729] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:00.148106 7560 consensus_queue.cc:260] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:00.148739 7560 ts_tablet_manager.cc:1398] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:00.232455 7492 raft_consensus.cc:482] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:00.232676 7492 raft_consensus.cc:504] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:00.233561 7543 raft_consensus.cc:482] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:00.233673 7492 leader_election.cc:288] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:00.233773 7543 raft_consensus.cc:504] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:00.234880 7543 leader_election.cc:288] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:00.236747 7154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a778d21259eb42e68c8b0fd534facc5b" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:00.236929 7154 raft_consensus.cc:2454] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:10:00.237334 7353 leader_election.cc:302] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:00.237668 7492 raft_consensus.cc:2781] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:00.237762 7492 raft_consensus.cc:482] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:00.237865 7492 raft_consensus.cc:3037] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:00.241287 7492 raft_consensus.cc:504] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:00.242575 7154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a778d21259eb42e68c8b0fd534facc5b" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:00.242830 7154 raft_consensus.cc:3037] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:00.245566 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a778d21259eb42e68c8b0fd534facc5b" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:00.245844 7154 raft_consensus.cc:2454] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:00.245921 7420 raft_consensus.cc:2379] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db0742ee3860468590eefef6ae6074fd in current term 1: Already voted for candidate b9957d4086bc4a76b18ed9265a5739d5 in this term.
I20240501 20:10:00.246295 7353 leader_election.cc:302] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:00.246739 7495 raft_consensus.cc:2781] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:00.247715 7495 raft_consensus.cc:686] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Becoming Leader. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Running, Role: LEADER
I20240501 20:10:00.248212 7495 consensus_queue.cc:237] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:00.250960 7492 leader_election.cc:288] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Requested vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:00.251852 4087 catalog_manager.cc:5484] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72). New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } }
W20240501 20:10:00.258879 4281 tablet.cc:2369] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:10:00.282416 4088 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240501 20:10:00.284337 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a778d21259eb42e68c8b0fd534facc5b" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:10:00.284624 4234 raft_consensus.cc:3037] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:00.287163 4234 raft_consensus.cc:2454] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:00.288128 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a778d21259eb42e68c8b0fd534facc5b" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:00.288446 4233 raft_consensus.cc:2379] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db0742ee3860468590eefef6ae6074fd in current term 1: Already voted for candidate b9957d4086bc4a76b18ed9265a5739d5 in this term.
I20240501 20:10:00.284394 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a778d21259eb42e68c8b0fd534facc5b" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:00.288939 4235 raft_consensus.cc:2362] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1. Re-sending same reply.
I20240501 20:10:00.289608 7090 leader_election.cc:302] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db0742ee3860468590eefef6ae6074fd; no voters: b9957d4086bc4a76b18ed9265a5739d5, dca57823003e4729b769ca12367d7530
I20240501 20:10:00.290074 7543 raft_consensus.cc:2726] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:00.290585 4088 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240501 20:10:00.301061 4088 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 63a07bfc290f4bd5a180562344518054
I20240501 20:10:00.301497 4088 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 54fd2e3ab852426abb0bdd7c3f1b8bfc
I20240501 20:10:00.301913 4088 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 4894528c9205457d8586934e822f4cff
I20240501 20:10:00.302027 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a07bfc290f4bd5a180562344518054 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:00 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:00.302234 4346 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a07bfc290f4bd5a180562344518054 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:00 UTC) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:00.302320 7519 tablet_replica.cc:330] T 63a07bfc290f4bd5a180562344518054 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:00.302534 7519 raft_consensus.cc:2229] T 63a07bfc290f4bd5a180562344518054 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:00.302927 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet 54fd2e3ab852426abb0bdd7c3f1b8bfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:00 UTC) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:00.302969 7519 raft_consensus.cc:2258] T 63a07bfc290f4bd5a180562344518054 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:00.303210 7514 tablet_replica.cc:330] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:00.303431 7514 raft_consensus.cc:2229] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:00.303750 7519 ts_tablet_manager.cc:1875] T 63a07bfc290f4bd5a180562344518054 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:00.303787 7514 raft_consensus.cc:2258] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:00.305003 7514 ts_tablet_manager.cc:1875] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:00.306763 7512 tablet_replica.cc:330] T 63a07bfc290f4bd5a180562344518054 P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:00.307049 7512 raft_consensus.cc:2229] T 63a07bfc290f4bd5a180562344518054 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:00.307307 7512 raft_consensus.cc:2258] T 63a07bfc290f4bd5a180562344518054 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:00.307662 4346 tablet_service.cc:1514] Processing DeleteTablet for tablet 54fd2e3ab852426abb0bdd7c3f1b8bfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:00 UTC) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:00.307900 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet 4894528c9205457d8586934e822f4cff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:00 UTC) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:00.308032 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet 4894528c9205457d8586934e822f4cff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:00 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:00.308166 7512 ts_tablet_manager.cc:1875] T 63a07bfc290f4bd5a180562344518054 P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:00.311739 7514 ts_tablet_manager.cc:1888] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:00.311926 7514 log.cc:1196] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/54fd2e3ab852426abb0bdd7c3f1b8bfc
I20240501 20:10:00.312310 7514 ts_tablet_manager.cc:1909] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 3e974c03d3c546f9ab6c91e762a5426b: Deleting consensus metadata
I20240501 20:10:00.313189 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet 54fd2e3ab852426abb0bdd7c3f1b8bfc (table rebalance_test_table_2 [id=22f5d6c17f44439eb5b61fdabb03f88f]) successfully deleted
I20240501 20:10:00.314937 4611 tablet_service.cc:1514] Processing DeleteTablet for tablet 63a07bfc290f4bd5a180562344518054 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:00 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:00.315104 4609 tablet_service.cc:1514] Processing DeleteTablet for tablet 54fd2e3ab852426abb0bdd7c3f1b8bfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:00 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:00.316462 7517 tablet_replica.cc:330] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:00.316679 7517 raft_consensus.cc:2229] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:00.317050 7517 raft_consensus.cc:2258] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:00.317797 7517 ts_tablet_manager.cc:1875] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:00.319097 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet 4894528c9205457d8586934e822f4cff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:00 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:00.332590 7512 ts_tablet_manager.cc:1888] T 63a07bfc290f4bd5a180562344518054 P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:00.332757 7512 log.cc:1196] T 63a07bfc290f4bd5a180562344518054 P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/63a07bfc290f4bd5a180562344518054
I20240501 20:10:00.333565 7517 ts_tablet_manager.cc:1888] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:00.333679 7517 log.cc:1196] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/63a07bfc290f4bd5a180562344518054
I20240501 20:10:00.334028 7517 ts_tablet_manager.cc:1909] T 63a07bfc290f4bd5a180562344518054 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:00.335088 7519 ts_tablet_manager.cc:1888] T 63a07bfc290f4bd5a180562344518054 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:00.335232 7519 log.cc:1196] T 63a07bfc290f4bd5a180562344518054 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/63a07bfc290f4bd5a180562344518054
I20240501 20:10:00.335687 7519 ts_tablet_manager.cc:1909] T 63a07bfc290f4bd5a180562344518054 P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:00.336668 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet 63a07bfc290f4bd5a180562344518054 (table rebalance_test_table_2 [id=22f5d6c17f44439eb5b61fdabb03f88f]) successfully deleted
I20240501 20:10:00.336764 7512 ts_tablet_manager.cc:1909] T 63a07bfc290f4bd5a180562344518054 P 10b47782558c41ee91f5e164a372049a: Deleting consensus metadata
I20240501 20:10:00.336671 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet 63a07bfc290f4bd5a180562344518054 (table rebalance_test_table_2 [id=22f5d6c17f44439eb5b61fdabb03f88f]) successfully deleted
I20240501 20:10:00.338160 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet 63a07bfc290f4bd5a180562344518054 (table rebalance_test_table_2 [id=22f5d6c17f44439eb5b61fdabb03f88f]) successfully deleted
I20240501 20:10:00.340847 7519 tablet_replica.cc:330] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:00.341553 7517 tablet_replica.cc:330] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:00.341837 7517 raft_consensus.cc:2229] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:00.342088 7517 raft_consensus.cc:2258] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:00.342772 7519 raft_consensus.cc:2229] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:00.342841 7517 ts_tablet_manager.cc:1875] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:00.343145 7519 raft_consensus.cc:2258] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:00.343887 7519 ts_tablet_manager.cc:1875] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:00.337769 7512 tablet_replica.cc:330] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:00.348819 7512 raft_consensus.cc:2229] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:00.349258 7512 raft_consensus.cc:2258] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:00.350814 7517 ts_tablet_manager.cc:1888] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:00.351022 7517 log.cc:1196] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/54fd2e3ab852426abb0bdd7c3f1b8bfc
I20240501 20:10:00.351678 7517 ts_tablet_manager.cc:1909] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:00.350900 7512 ts_tablet_manager.cc:1875] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:00.353155 7517 tablet_replica.cc:330] T 4894528c9205457d8586934e822f4cff P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:00.353085 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet 54fd2e3ab852426abb0bdd7c3f1b8bfc (table rebalance_test_table_2 [id=22f5d6c17f44439eb5b61fdabb03f88f]) successfully deleted
I20240501 20:10:00.353379 7517 raft_consensus.cc:2229] T 4894528c9205457d8586934e822f4cff P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:00.353654 7517 raft_consensus.cc:2258] T 4894528c9205457d8586934e822f4cff P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:00.354518 7517 ts_tablet_manager.cc:1875] T 4894528c9205457d8586934e822f4cff P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:00.377780 7519 ts_tablet_manager.cc:1888] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:00.379420 7512 ts_tablet_manager.cc:1888] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:00.379781 7512 log.cc:1196] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/54fd2e3ab852426abb0bdd7c3f1b8bfc
I20240501 20:10:00.380450 7512 ts_tablet_manager.cc:1909] T 54fd2e3ab852426abb0bdd7c3f1b8bfc P 10b47782558c41ee91f5e164a372049a: Deleting consensus metadata
I20240501 20:10:00.381949 7512 tablet_replica.cc:330] T 4894528c9205457d8586934e822f4cff P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:00.382010 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet 54fd2e3ab852426abb0bdd7c3f1b8bfc (table rebalance_test_table_2 [id=22f5d6c17f44439eb5b61fdabb03f88f]) successfully deleted
I20240501 20:10:00.382359 7519 log.cc:1196] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/4894528c9205457d8586934e822f4cff
I20240501 20:10:00.383123 7519 ts_tablet_manager.cc:1909] T 4894528c9205457d8586934e822f4cff P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:00.385921 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet 4894528c9205457d8586934e822f4cff (table rebalance_test_table_2 [id=22f5d6c17f44439eb5b61fdabb03f88f]) successfully deleted
I20240501 20:10:00.394703 7512 raft_consensus.cc:2229] T 4894528c9205457d8586934e822f4cff P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:00.395254 7512 raft_consensus.cc:2258] T 4894528c9205457d8586934e822f4cff P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:00.395679 7517 ts_tablet_manager.cc:1888] T 4894528c9205457d8586934e822f4cff P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:00.395839 7517 log.cc:1196] T 4894528c9205457d8586934e822f4cff P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/4894528c9205457d8586934e822f4cff
I20240501 20:10:00.396327 7517 ts_tablet_manager.cc:1909] T 4894528c9205457d8586934e822f4cff P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:00.396710 7512 ts_tablet_manager.cc:1875] T 4894528c9205457d8586934e822f4cff P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:00.397908 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet 4894528c9205457d8586934e822f4cff (table rebalance_test_table_2 [id=22f5d6c17f44439eb5b61fdabb03f88f]) successfully deleted
I20240501 20:10:00.398085 4087 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:00.403731 7512 ts_tablet_manager.cc:1888] T 4894528c9205457d8586934e822f4cff P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:00.403915 7512 log.cc:1196] T 4894528c9205457d8586934e822f4cff P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/4894528c9205457d8586934e822f4cff
I20240501 20:10:00.404525 7512 ts_tablet_manager.cc:1909] T 4894528c9205457d8586934e822f4cff P 10b47782558c41ee91f5e164a372049a: Deleting consensus metadata
I20240501 20:10:00.407743 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet 4894528c9205457d8586934e822f4cff (table rebalance_test_table_2 [id=22f5d6c17f44439eb5b61fdabb03f88f]) successfully deleted
I20240501 20:10:00.408177 7154 raft_consensus.cc:1261] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:00.408964 4235 raft_consensus.cc:1261] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:00.410140 7492 consensus_queue.cc:1035] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:00.410452 7495 consensus_queue.cc:1035] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:00.428943 4610 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:10:00.431597 7543 consensus_queue.cc:1035] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:00.434034 4347 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:00.445736 7543 consensus_queue.cc:1035] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:00.449330 7545 consensus_queue.cc:1035] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:00.469683 7543 consensus_queue.cc:1035] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:00.515988 4478 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:00.607329 7401 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:00.613003 7267 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:00.615079 4088 catalog_manager.cc:5576] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321) does not have the latest schema for tablet a778d21259eb42e68c8b0fd534facc5b (table rebalancer_dynamic_table [id=c133c6c195dc43b6891545bfdf601e5d]). Expected version 1 got 0
I20240501 20:10:00.619690 7133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:00.647709 7543 consensus_queue.cc:1035] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:00.647811 4091 catalog_manager.cc:5576] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941) does not have the latest schema for tablet a778d21259eb42e68c8b0fd534facc5b (table rebalancer_dynamic_table [id=c133c6c195dc43b6891545bfdf601e5d]). Expected version 1 got 0
I20240501 20:10:00.649613 4087 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:00.652302 7622 tablet.cc:1722] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:00.668455 4091 catalog_manager.cc:5576] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941) does not have the latest schema for tablet a778d21259eb42e68c8b0fd534facc5b (table rebalancer_dynamic_table [id=c133c6c195dc43b6891545bfdf601e5d]). Expected version 1 got 0
I20240501 20:10:00.673099 7619 tablet.cc:1722] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:00.675076 7617 consensus_queue.cc:1035] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:00.675652 7619 tablet.cc:1620] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:00.680830 7002 tablet_service.cc:1467] Processing CreateTablet for tablet 3a982d9467a14f88a54a240266be728d (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=983618faca8342e4aea12a20efab90c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:00.680919 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 53be6ced1df34676b1a34bc5cfe1ac87 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=983618faca8342e4aea12a20efab90c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:00.681305 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a982d9467a14f88a54a240266be728d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:00.681411 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53be6ced1df34676b1a34bc5cfe1ac87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:00.683105 7001 tablet_service.cc:1467] Processing CreateTablet for tablet 3809cc8ffe6b4a9e99ebbb9d26a65279 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=983618faca8342e4aea12a20efab90c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:00.683583 7001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3809cc8ffe6b4a9e99ebbb9d26a65279. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:00.687712 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 53be6ced1df34676b1a34bc5cfe1ac87 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=983618faca8342e4aea12a20efab90c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:00.688221 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53be6ced1df34676b1a34bc5cfe1ac87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:00.688805 4347 tablet_service.cc:1467] Processing CreateTablet for tablet 3a982d9467a14f88a54a240266be728d (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=983618faca8342e4aea12a20efab90c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:00.689311 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a982d9467a14f88a54a240266be728d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:00.691009 4346 tablet_service.cc:1467] Processing CreateTablet for tablet 3809cc8ffe6b4a9e99ebbb9d26a65279 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=983618faca8342e4aea12a20efab90c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:00.691465 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3809cc8ffe6b4a9e99ebbb9d26a65279. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:00.699878 4091 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=c133c6c195dc43b6891545bfdf601e5d] alter complete (version 1)
I20240501 20:10:00.705677 7628 tablet_bootstrap.cc:492] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:00.711907 7133 tablet_service.cc:1467] Processing CreateTablet for tablet 3a982d9467a14f88a54a240266be728d (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=983618faca8342e4aea12a20efab90c0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:00.712467 7133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a982d9467a14f88a54a240266be728d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:00.720382 7267 tablet_service.cc:1467] Processing CreateTablet for tablet 3809cc8ffe6b4a9e99ebbb9d26a65279 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=983618faca8342e4aea12a20efab90c0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:00.720914 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3809cc8ffe6b4a9e99ebbb9d26a65279. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:00.722290 7266 tablet_service.cc:1467] Processing CreateTablet for tablet 53be6ced1df34676b1a34bc5cfe1ac87 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=983618faca8342e4aea12a20efab90c0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:00.722788 7266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53be6ced1df34676b1a34bc5cfe1ac87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:00.743144 7633 tablet_bootstrap.cc:492] T 53be6ced1df34676b1a34bc5cfe1ac87 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:00.747037 7631 tablet_bootstrap.cc:492] T 3a982d9467a14f88a54a240266be728d P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:00.748803 7631 tablet_bootstrap.cc:654] T 3a982d9467a14f88a54a240266be728d P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:00.750345 7633 tablet_bootstrap.cc:654] T 53be6ced1df34676b1a34bc5cfe1ac87 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:00.751108 7632 tablet_bootstrap.cc:492] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:00.753473 7632 tablet_bootstrap.cc:654] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:00.768644 7633 tablet_bootstrap.cc:492] T 53be6ced1df34676b1a34bc5cfe1ac87 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:00.768699 7622 tablet.cc:1620] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530: MemRowSet was empty: no flush needed.
I20240501 20:10:00.768885 7633 ts_tablet_manager.cc:1367] T 53be6ced1df34676b1a34bc5cfe1ac87 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.002s
I20240501 20:10:00.776858 7633 raft_consensus.cc:348] T 53be6ced1df34676b1a34bc5cfe1ac87 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:00.777123 7633 raft_consensus.cc:374] T 53be6ced1df34676b1a34bc5cfe1ac87 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:00.777204 7633 raft_consensus.cc:729] T 53be6ced1df34676b1a34bc5cfe1ac87 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:00.777585 7633 consensus_queue.cc:260] T 53be6ced1df34676b1a34bc5cfe1ac87 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:00.778191 7633 ts_tablet_manager.cc:1398] T 53be6ced1df34676b1a34bc5cfe1ac87 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240501 20:10:00.780666 7627 tablet_bootstrap.cc:492] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:00.784369 7628 tablet_bootstrap.cc:654] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:00.785146 7632 tablet_bootstrap.cc:492] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:00.785310 7632 ts_tablet_manager.cc:1367] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20240501 20:10:00.786114 7632 raft_consensus.cc:348] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:00.786322 7632 raft_consensus.cc:374] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:00.786402 7632 raft_consensus.cc:729] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:00.786700 7632 consensus_queue.cc:260] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:00.787235 7632 ts_tablet_manager.cc:1398] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:10:00.788247 7631 tablet_bootstrap.cc:492] T 3a982d9467a14f88a54a240266be728d P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:00.788408 7631 ts_tablet_manager.cc:1367] T 3a982d9467a14f88a54a240266be728d P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.000s
I20240501 20:10:00.789358 7631 raft_consensus.cc:348] T 3a982d9467a14f88a54a240266be728d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:00.789647 7631 raft_consensus.cc:374] T 3a982d9467a14f88a54a240266be728d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:00.789816 7631 raft_consensus.cc:729] T 3a982d9467a14f88a54a240266be728d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:00.790199 7631 consensus_queue.cc:260] T 3a982d9467a14f88a54a240266be728d P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:00.790926 7631 ts_tablet_manager.cc:1398] T 3a982d9467a14f88a54a240266be728d P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:00.795383 7627 tablet_bootstrap.cc:654] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:00.798244 4215 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 20:10:00.798696 7626 tablet_bootstrap.cc:492] T 3a982d9467a14f88a54a240266be728d P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:00.800904 7626 tablet_bootstrap.cc:654] T 3a982d9467a14f88a54a240266be728d P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:00.802868 7632 tablet_bootstrap.cc:492] T 53be6ced1df34676b1a34bc5cfe1ac87 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:00.802913 7623 tablet.cc:1722] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:00.803681 7623 tablet.cc:1620] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:10:00.804608 7632 tablet_bootstrap.cc:654] T 53be6ced1df34676b1a34bc5cfe1ac87 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:00.807729 4091 catalog_manager.cc:5576] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321) does not have the latest schema for tablet a778d21259eb42e68c8b0fd534facc5b (table rebalancer_dynamic_table [id=c133c6c195dc43b6891545bfdf601e5d]). Expected version 1 got 0
I20240501 20:10:00.817054 7628 tablet_bootstrap.cc:492] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:10:00.817283 7628 ts_tablet_manager.cc:1367] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.112s user 0.004s sys 0.004s
I20240501 20:10:00.818131 7628 raft_consensus.cc:348] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:00.818495 7628 raft_consensus.cc:374] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:00.818684 7628 raft_consensus.cc:729] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:10:00.818992 7628 consensus_queue.cc:260] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:00.819708 7628 ts_tablet_manager.cc:1398] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:00.824287 7627 tablet_bootstrap.cc:492] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:00.824443 7627 ts_tablet_manager.cc:1367] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.044s user 0.007s sys 0.000s
I20240501 20:10:00.825371 7627 raft_consensus.cc:348] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:00.825673 7627 raft_consensus.cc:374] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:00.825834 7627 raft_consensus.cc:729] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:00.826196 7627 consensus_queue.cc:260] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:00.826911 7628 tablet_bootstrap.cc:492] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:00.827872 7002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:00.830150 4091 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240501 20:10:00.830456 4091 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240501 20:10:00.830585 7627 ts_tablet_manager.cc:1398] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:10:00.836028 4091 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet b72266b07e89450ca96948dd36351dd2
I20240501 20:10:00.837466 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet b72266b07e89450ca96948dd36351dd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:00 UTC) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:00.838559 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet b72266b07e89450ca96948dd36351dd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:00 UTC) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:00.838966 7512 tablet_replica.cc:330] T b72266b07e89450ca96948dd36351dd2 P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:00.839274 7512 raft_consensus.cc:2229] T b72266b07e89450ca96948dd36351dd2 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:00.840092 7628 tablet_bootstrap.cc:654] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:00.840130 7512 raft_consensus.cc:2258] T b72266b07e89450ca96948dd36351dd2 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:00.841267 7512 ts_tablet_manager.cc:1875] T b72266b07e89450ca96948dd36351dd2 P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:00.844695 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet b72266b07e89450ca96948dd36351dd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:00 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:00.844985 7519 tablet_replica.cc:330] T b72266b07e89450ca96948dd36351dd2 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:00.844995 4091 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 1ba181e4f1094e63adf459bf12c7f464
I20240501 20:10:00.845188 7519 raft_consensus.cc:2229] T b72266b07e89450ca96948dd36351dd2 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:00.845427 7519 raft_consensus.cc:2258] T b72266b07e89450ca96948dd36351dd2 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:00.845678 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ba181e4f1094e63adf459bf12c7f464 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:00 UTC) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:00.846184 7519 ts_tablet_manager.cc:1875] T b72266b07e89450ca96948dd36351dd2 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:00.846153 4090 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:00.850330 7512 ts_tablet_manager.cc:1888] T b72266b07e89450ca96948dd36351dd2 P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:00.850594 7512 log.cc:1196] T b72266b07e89450ca96948dd36351dd2 P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/b72266b07e89450ca96948dd36351dd2
I20240501 20:10:00.851446 7512 ts_tablet_manager.cc:1909] T b72266b07e89450ca96948dd36351dd2 P 10b47782558c41ee91f5e164a372049a: Deleting consensus metadata
I20240501 20:10:00.852473 7512 tablet_replica.cc:330] T 1ba181e4f1094e63adf459bf12c7f464 P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:00.852543 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet b72266b07e89450ca96948dd36351dd2 (table rebalance_test_table_4 [id=2287807e30d84209930d192b6b5a3d78]) successfully deleted
I20240501 20:10:00.852699 7512 raft_consensus.cc:2229] T 1ba181e4f1094e63adf459bf12c7f464 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:00.852957 7512 raft_consensus.cc:2258] T 1ba181e4f1094e63adf459bf12c7f464 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:00.853461 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ba181e4f1094e63adf459bf12c7f464 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:00 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:00.853711 7512 ts_tablet_manager.cc:1875] T 1ba181e4f1094e63adf459bf12c7f464 P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:00.853960 4091 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 35c0930b782a47c2a74cca36c261a08d
I20240501 20:10:00.854327 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ba181e4f1094e63adf459bf12c7f464 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:00 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:00.854583 7517 tablet_replica.cc:330] T 1ba181e4f1094e63adf459bf12c7f464 P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:00.854759 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet 35c0930b782a47c2a74cca36c261a08d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:00 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:00.854857 7517 raft_consensus.cc:2229] T 1ba181e4f1094e63adf459bf12c7f464 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:00.855093 7517 raft_consensus.cc:2258] T 1ba181e4f1094e63adf459bf12c7f464 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:00.855805 7517 ts_tablet_manager.cc:1875] T 1ba181e4f1094e63adf459bf12c7f464 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:00.857364 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet 35c0930b782a47c2a74cca36c261a08d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:00 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:00.859884 4477 tablet_service.cc:1514] Processing DeleteTablet for tablet 35c0930b782a47c2a74cca36c261a08d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:00 UTC) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:00.864100 7517 ts_tablet_manager.cc:1888] T 1ba181e4f1094e63adf459bf12c7f464 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:00.864274 7517 log.cc:1196] T 1ba181e4f1094e63adf459bf12c7f464 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/1ba181e4f1094e63adf459bf12c7f464
I20240501 20:10:00.864742 7517 ts_tablet_manager.cc:1909] T 1ba181e4f1094e63adf459bf12c7f464 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:00.865480 7641 tablet_replica.cc:330] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:00.865800 7641 raft_consensus.cc:2229] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:00.865949 7517 tablet_replica.cc:330] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:00.866178 7641 raft_consensus.cc:2258] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:00.866204 7517 raft_consensus.cc:2229] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:00.866703 7517 raft_consensus.cc:2258] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:00.866981 7641 ts_tablet_manager.cc:1875] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:00.867597 7517 ts_tablet_manager.cc:1875] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:00.867698 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet 1ba181e4f1094e63adf459bf12c7f464 (table rebalance_test_table_4 [id=2287807e30d84209930d192b6b5a3d78]) successfully deleted
I20240501 20:10:00.871330 7622 tablet.cc:1722] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:00.871910 7622 tablet.cc:1620] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530: MemRowSet was empty: no flush needed.
I20240501 20:10:00.872160 7626 tablet_bootstrap.cc:492] T 3a982d9467a14f88a54a240266be728d P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:00.872416 7626 ts_tablet_manager.cc:1367] T 3a982d9467a14f88a54a240266be728d P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.000s
I20240501 20:10:00.873754 7626 raft_consensus.cc:348] T 3a982d9467a14f88a54a240266be728d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:00.874204 7626 raft_consensus.cc:374] T 3a982d9467a14f88a54a240266be728d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:00.874392 7626 raft_consensus.cc:729] T 3a982d9467a14f88a54a240266be728d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:00.874881 7626 consensus_queue.cc:260] T 3a982d9467a14f88a54a240266be728d P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:00.877599 7626 ts_tablet_manager.cc:1398] T 3a982d9467a14f88a54a240266be728d P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:10:00.879331 7626 tablet_bootstrap.cc:492] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:00.880543 7538 raft_consensus.cc:482] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:00.880717 7538 raft_consensus.cc:504] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:00.881700 7538 leader_election.cc:288] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:00.884485 7628 tablet_bootstrap.cc:492] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:10:00.884795 7628 ts_tablet_manager.cc:1367] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.006s
I20240501 20:10:00.886242 4090 catalog_manager.cc:5576] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321) does not have the latest schema for tablet a778d21259eb42e68c8b0fd534facc5b (table rebalancer_dynamic_table [id=c133c6c195dc43b6891545bfdf601e5d]). Expected version 2 got 1
I20240501 20:10:00.886837 7628 raft_consensus.cc:348] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:00.887085 7628 raft_consensus.cc:374] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:00.887167 7628 raft_consensus.cc:729] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:10:00.887493 7628 consensus_queue.cc:260] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:00.888366 7628 ts_tablet_manager.cc:1398] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:10:00.892652 4090 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=c133c6c195dc43b6891545bfdf601e5d] alter complete (version 2)
I20240501 20:10:00.893290 7517 ts_tablet_manager.cc:1888] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:00.893617 7517 log.cc:1196] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/35c0930b782a47c2a74cca36c261a08d
I20240501 20:10:00.894331 7517 ts_tablet_manager.cc:1909] T 35c0930b782a47c2a74cca36c261a08d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:00.899288 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet 35c0930b782a47c2a74cca36c261a08d (table rebalance_test_table_4 [id=2287807e30d84209930d192b6b5a3d78]) successfully deleted
I20240501 20:10:00.903126 7512 ts_tablet_manager.cc:1888] T 1ba181e4f1094e63adf459bf12c7f464 P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:00.904145 7512 log.cc:1196] T 1ba181e4f1094e63adf459bf12c7f464 P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/1ba181e4f1094e63adf459bf12c7f464
I20240501 20:10:00.905465 7626 tablet_bootstrap.cc:654] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:00.907001 7512 ts_tablet_manager.cc:1909] T 1ba181e4f1094e63adf459bf12c7f464 P 10b47782558c41ee91f5e164a372049a: Deleting consensus metadata
I20240501 20:10:00.913599 7154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a982d9467a14f88a54a240266be728d" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:00.913892 7154 raft_consensus.cc:2454] T 3a982d9467a14f88a54a240266be728d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:10:00.914551 4299 leader_election.cc:302] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:00.915056 7537 raft_consensus.cc:482] T 3a982d9467a14f88a54a240266be728d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:00.915235 7537 raft_consensus.cc:504] T 3a982d9467a14f88a54a240266be728d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:00.915757 7632 tablet_bootstrap.cc:492] T 53be6ced1df34676b1a34bc5cfe1ac87 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:00.915992 7632 ts_tablet_manager.cc:1367] T 53be6ced1df34676b1a34bc5cfe1ac87 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.113s user 0.004s sys 0.013s
I20240501 20:10:00.916208 7538 raft_consensus.cc:2781] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:00.916325 7538 raft_consensus.cc:482] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:00.916414 7538 raft_consensus.cc:3037] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:00.916901 7519 ts_tablet_manager.cc:1888] T b72266b07e89450ca96948dd36351dd2 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:00.916869 7632 raft_consensus.cc:348] T 53be6ced1df34676b1a34bc5cfe1ac87 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:00.917038 7519 log.cc:1196] T b72266b07e89450ca96948dd36351dd2 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/b72266b07e89450ca96948dd36351dd2
I20240501 20:10:00.917083 7632 raft_consensus.cc:374] T 53be6ced1df34676b1a34bc5cfe1ac87 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:00.917150 7632 raft_consensus.cc:729] T 53be6ced1df34676b1a34bc5cfe1ac87 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:00.917635 7632 consensus_queue.cc:260] T 53be6ced1df34676b1a34bc5cfe1ac87 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:00.918202 7632 ts_tablet_manager.cc:1398] T 53be6ced1df34676b1a34bc5cfe1ac87 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:10:00.920536 7538 raft_consensus.cc:504] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:00.921288 7538 leader_election.cc:288] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Requested vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:00.922078 7154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a982d9467a14f88a54a240266be728d" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:00.922326 7154 raft_consensus.cc:3037] T 3a982d9467a14f88a54a240266be728d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:10:00.922353 4414 tablet.cc:2369] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 10b47782558c41ee91f5e164a372049a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:10:00.925143 7154 raft_consensus.cc:2454] T 3a982d9467a14f88a54a240266be728d P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:10:00.924948 7619 tablet.cc:1722] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:00.925885 4299 leader_election.cc:302] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:00.925964 7619 tablet.cc:1620] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:00.926348 7538 raft_consensus.cc:2781] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:00.926713 7538 raft_consensus.cc:686] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Becoming Leader. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Running, Role: LEADER
I20240501 20:10:00.927289 7538 consensus_queue.cc:237] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:00.928627 7623 tablet.cc:1722] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:00.929672 7623 tablet.cc:1620] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:10:00.935053 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet 1ba181e4f1094e63adf459bf12c7f464 (table rebalance_test_table_4 [id=2287807e30d84209930d192b6b5a3d78]) successfully deleted
I20240501 20:10:00.935660 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a982d9467a14f88a54a240266be728d" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:00.935693 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a982d9467a14f88a54a240266be728d" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:00.935909 7020 raft_consensus.cc:2454] T 3a982d9467a14f88a54a240266be728d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:10:00.939343 7537 leader_election.cc:288] T 3a982d9467a14f88a54a240266be728d P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:00.948982 7205 raft_consensus.cc:482] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:00.949229 7205 raft_consensus.cc:504] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:00.950500 7205 leader_election.cc:288] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
W20240501 20:10:00.952682 4170 consensus_peers.cc:485] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530 -> Peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Couldn't send request to peer 1d34e18a1d394d3f91c9f50d1c7bea4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ba181e4f1094e63adf459bf12c7f464. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:00.954056 7519 ts_tablet_manager.cc:1909] T b72266b07e89450ca96948dd36351dd2 P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:00.957234 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet b72266b07e89450ca96948dd36351dd2 (table rebalance_test_table_4 [id=2287807e30d84209930d192b6b5a3d78]) successfully deleted
I20240501 20:10:00.958055 7519 tablet_replica.cc:330] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:00.959233 7519 raft_consensus.cc:2229] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:00.959980 7519 raft_consensus.cc:2258] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:00.964028 7519 ts_tablet_manager.cc:1875] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:00.972314 7626 tablet_bootstrap.cc:492] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:00.973031 7626 ts_tablet_manager.cc:1367] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.094s user 0.000s sys 0.007s
I20240501 20:10:00.977875 7626 raft_consensus.cc:348] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:00.978260 7626 raft_consensus.cc:374] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:00.978394 7626 raft_consensus.cc:729] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:00.980511 4091 catalog_manager.cc:5484] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a reported cstate change: term changed from 0 to 1, leader changed from <none> to 10b47782558c41ee91f5e164a372049a (127.29.219.66). New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:00.981950 7626 consensus_queue.cc:260] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:00.988039 7641 ts_tablet_manager.cc:1888] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:00.990852 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53be6ced1df34676b1a34bc5cfe1ac87" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:00.991114 7420 raft_consensus.cc:2454] T 53be6ced1df34676b1a34bc5cfe1ac87 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:00.991662 4170 leader_election.cc:302] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:00.991961 7205 raft_consensus.cc:2781] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:00.991971 7626 ts_tablet_manager.cc:1398] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240501 20:10:00.992076 7205 raft_consensus.cc:482] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:00.992154 7205 raft_consensus.cc:3037] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:00.994139 7641 log.cc:1196] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/b72266b07e89450ca96948dd36351dd2
I20240501 20:10:00.994045 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53be6ced1df34676b1a34bc5cfe1ac87" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:00.994331 7286 raft_consensus.cc:2454] T 53be6ced1df34676b1a34bc5cfe1ac87 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:00.994879 7641 ts_tablet_manager.cc:1909] T b72266b07e89450ca96948dd36351dd2 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting consensus metadata
I20240501 20:10:00.995075 7205 raft_consensus.cc:504] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:00.995921 7641 tablet_replica.cc:330] T 35c0930b782a47c2a74cca36c261a08d P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:00.996109 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet b72266b07e89450ca96948dd36351dd2 (table rebalance_test_table_4 [id=2287807e30d84209930d192b6b5a3d78]) successfully deleted
I20240501 20:10:00.996155 7641 raft_consensus.cc:2229] T 35c0930b782a47c2a74cca36c261a08d P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:00.996394 7641 raft_consensus.cc:2258] T 35c0930b782a47c2a74cca36c261a08d P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:00.996392 7205 leader_election.cc:288] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:00.997066 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53be6ced1df34676b1a34bc5cfe1ac87" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:00.997313 7420 raft_consensus.cc:3037] T 53be6ced1df34676b1a34bc5cfe1ac87 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:00.997591 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53be6ced1df34676b1a34bc5cfe1ac87" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:00.997835 7286 raft_consensus.cc:3037] T 53be6ced1df34676b1a34bc5cfe1ac87 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:01.000300 7420 raft_consensus.cc:2454] T 53be6ced1df34676b1a34bc5cfe1ac87 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:01.000743 4170 leader_election.cc:302] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:01.000852 7286 raft_consensus.cc:2454] T 53be6ced1df34676b1a34bc5cfe1ac87 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:01.001192 7205 raft_consensus.cc:2781] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:01.001503 7205 raft_consensus.cc:686] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:10:01.001914 7205 consensus_queue.cc:237] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:01.003190 7519 ts_tablet_manager.cc:1888] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:01.003342 7519 log.cc:1196] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/1ba181e4f1094e63adf459bf12c7f464
I20240501 20:10:01.003911 7519 ts_tablet_manager.cc:1909] T 1ba181e4f1094e63adf459bf12c7f464 P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:01.005136 7519 tablet_replica.cc:330] T 35c0930b782a47c2a74cca36c261a08d P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:01.005429 7519 raft_consensus.cc:2229] T 35c0930b782a47c2a74cca36c261a08d P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:01.005751 7519 raft_consensus.cc:2258] T 35c0930b782a47c2a74cca36c261a08d P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:01.005812 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet 1ba181e4f1094e63adf459bf12c7f464 (table rebalance_test_table_4 [id=2287807e30d84209930d192b6b5a3d78]) successfully deleted
I20240501 20:10:01.006711 7519 ts_tablet_manager.cc:1875] T 35c0930b782a47c2a74cca36c261a08d P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:01.011304 7641 ts_tablet_manager.cc:1875] T 35c0930b782a47c2a74cca36c261a08d P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:01.011524 4367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a982d9467a14f88a54a240266be728d" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:10:01.017978 4091 catalog_manager.cc:5484] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:01.019315 4088 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:01.024376 7154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a982d9467a14f88a54a240266be728d" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:01.024749 7154 raft_consensus.cc:2379] T 3a982d9467a14f88a54a240266be728d P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef0a4ada548446b3a531cb39a3c4e38d in current term 1: Already voted for candidate 10b47782558c41ee91f5e164a372049a in this term.
I20240501 20:10:01.026044 6955 leader_election.cc:302] T 3a982d9467a14f88a54a240266be728d P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef0a4ada548446b3a531cb39a3c4e38d; no voters: 10b47782558c41ee91f5e164a372049a, db0742ee3860468590eefef6ae6074fd
I20240501 20:10:01.027834 7537 raft_consensus.cc:3037] T 3a982d9467a14f88a54a240266be728d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:01.028255 7641 ts_tablet_manager.cc:1888] T 35c0930b782a47c2a74cca36c261a08d P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:01.028409 7641 log.cc:1196] T 35c0930b782a47c2a74cca36c261a08d P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/35c0930b782a47c2a74cca36c261a08d
I20240501 20:10:01.028898 7641 ts_tablet_manager.cc:1909] T 35c0930b782a47c2a74cca36c261a08d P 3e974c03d3c546f9ab6c91e762a5426b: Deleting consensus metadata
I20240501 20:10:01.030859 7618 raft_consensus.cc:482] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:01.031023 7537 raft_consensus.cc:2726] T 3a982d9467a14f88a54a240266be728d P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:01.031042 7618 raft_consensus.cc:504] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:01.031380 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet 35c0930b782a47c2a74cca36c261a08d (table rebalance_test_table_4 [id=2287807e30d84209930d192b6b5a3d78]) successfully deleted
I20240501 20:10:01.031940 7618 leader_election.cc:288] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:01.033871 7519 ts_tablet_manager.cc:1888] T 35c0930b782a47c2a74cca36c261a08d P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:01.034075 7519 log.cc:1196] T 35c0930b782a47c2a74cca36c261a08d P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/35c0930b782a47c2a74cca36c261a08d
I20240501 20:10:01.034576 7519 ts_tablet_manager.cc:1909] T 35c0930b782a47c2a74cca36c261a08d P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:01.035636 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet 35c0930b782a47c2a74cca36c261a08d (table rebalance_test_table_4 [id=2287807e30d84209930d192b6b5a3d78]) successfully deleted
I20240501 20:10:01.035802 7619 tablet.cc:1722] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:01.040318 7622 tablet.cc:1722] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:01.041388 7537 raft_consensus.cc:482] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:01.041615 7537 raft_consensus.cc:504] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:01.042637 7537 leader_election.cc:288] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:01.044081 7623 tablet.cc:1722] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:01.057812 4367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3809cc8ffe6b4a9e99ebbb9d26a65279" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:10:01.058070 4367 raft_consensus.cc:2454] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:01.058689 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3809cc8ffe6b4a9e99ebbb9d26a65279" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:01.058910 7022 raft_consensus.cc:2454] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:01.059453 6957 leader_election.cc:302] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:01.059962 7222 leader_election.cc:302] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:01.061093 7537 raft_consensus.cc:2781] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:01.061192 7537 raft_consensus.cc:482] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:01.061309 7537 raft_consensus.cc:3037] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:01.063446 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3809cc8ffe6b4a9e99ebbb9d26a65279" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:01.063768 7286 raft_consensus.cc:2454] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:01.065469 7537 raft_consensus.cc:504] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:01.067045 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3809cc8ffe6b4a9e99ebbb9d26a65279" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:10:01.067306 4365 raft_consensus.cc:2454] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:01.068209 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3809cc8ffe6b4a9e99ebbb9d26a65279" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:01.068430 7286 raft_consensus.cc:3037] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:01.070446 7286 raft_consensus.cc:2454] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:01.070923 6956 leader_election.cc:302] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:01.073724 7537 leader_election.cc:288] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:01.074223 7618 raft_consensus.cc:2741] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240501 20:10:01.074668 7537 raft_consensus.cc:2781] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:01.075048 4090 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table_new_name [id=c133c6c195dc43b6891545bfdf601e5d] alter complete (version 3)
I20240501 20:10:01.078699 7537 raft_consensus.cc:686] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Becoming Leader. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Running, Role: LEADER
I20240501 20:10:01.079322 7537 consensus_queue.cc:237] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:01.081924 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3809cc8ffe6b4a9e99ebbb9d26a65279" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:10:01.082149 4365 raft_consensus.cc:3037] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:01.084218 4090 catalog_manager.cc:5484] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69). New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:01.084981 4365 raft_consensus.cc:2454] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:01.177657 4090 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:01.177939 4090 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:01.181926 4090 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet a778d21259eb42e68c8b0fd534facc5b
I20240501 20:10:01.182767 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet a778d21259eb42e68c8b0fd534facc5b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:01 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:01.183039 7519 tablet_replica.cc:330] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:01.183179 7133 tablet_service.cc:1514] Processing DeleteTablet for tablet a778d21259eb42e68c8b0fd534facc5b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:01 UTC) from {username='slave'} at 127.0.0.1:52036
I20240501 20:10:01.183248 7519 raft_consensus.cc:2229] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:01.183450 7519 raft_consensus.cc:2258] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:01.183530 7401 tablet_service.cc:1514] Processing DeleteTablet for tablet a778d21259eb42e68c8b0fd534facc5b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:01 UTC) from {username='slave'} at 127.0.0.1:40182
I20240501 20:10:01.184535 7679 tablet_replica.cc:330] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd: stopping tablet replica
I20240501 20:10:01.184526 7519 ts_tablet_manager.cc:1875] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:01.184891 7679 raft_consensus.cc:2229] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:01.185293 7679 raft_consensus.cc:2258] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:01.186187 7679 ts_tablet_manager.cc:1875] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:01.190162 7680 tablet_replica.cc:330] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5: stopping tablet replica
I20240501 20:10:01.190407 7680 raft_consensus.cc:2229] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:01.190819 7680 raft_consensus.cc:2258] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:01.191414 7519 ts_tablet_manager.cc:1888] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:01.191573 7519 log.cc:1196] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/a778d21259eb42e68c8b0fd534facc5b
I20240501 20:10:01.192018 7519 ts_tablet_manager.cc:1909] T a778d21259eb42e68c8b0fd534facc5b P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:01.193922 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet a778d21259eb42e68c8b0fd534facc5b (table rebalancer_dynamic_table_new_name [id=c133c6c195dc43b6891545bfdf601e5d]) successfully deleted
I20240501 20:10:01.199506 7680 ts_tablet_manager.cc:1875] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:01.201833 7679 ts_tablet_manager.cc:1888] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:01.201975 7679 log.cc:1196] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/wals/a778d21259eb42e68c8b0fd534facc5b
I20240501 20:10:01.207464 7680 ts_tablet_manager.cc:1888] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:01.207667 7680 log.cc:1196] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/wals/a778d21259eb42e68c8b0fd534facc5b
I20240501 20:10:01.208276 7680 ts_tablet_manager.cc:1909] T a778d21259eb42e68c8b0fd534facc5b P b9957d4086bc4a76b18ed9265a5739d5: Deleting consensus metadata
I20240501 20:10:01.209894 7679 ts_tablet_manager.cc:1909] T a778d21259eb42e68c8b0fd534facc5b P db0742ee3860468590eefef6ae6074fd: Deleting consensus metadata
I20240501 20:10:01.211817 4076 catalog_manager.cc:4830] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): tablet a778d21259eb42e68c8b0fd534facc5b (table rebalancer_dynamic_table_new_name [id=c133c6c195dc43b6891545bfdf601e5d]) successfully deleted
I20240501 20:10:01.217494 4079 catalog_manager.cc:4830] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): tablet a778d21259eb42e68c8b0fd534facc5b (table rebalancer_dynamic_table_new_name [id=c133c6c195dc43b6891545bfdf601e5d]) successfully deleted
I20240501 20:10:01.290292 4088 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:01.299429 7002 tablet_service.cc:1467] Processing CreateTablet for tablet 7f89f8faca5b4ec7918bdbf9ef9528a6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=da2afc94765b4977ad891669f4f8ede2]), partition=
I20240501 20:10:01.299907 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f89f8faca5b4ec7918bdbf9ef9528a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:01.303929 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 7f89f8faca5b4ec7918bdbf9ef9528a6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=da2afc94765b4977ad891669f4f8ede2]), partition=
I20240501 20:10:01.304107 7266 tablet_service.cc:1467] Processing CreateTablet for tablet 7f89f8faca5b4ec7918bdbf9ef9528a6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=da2afc94765b4977ad891669f4f8ede2]), partition=
I20240501 20:10:01.304308 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f89f8faca5b4ec7918bdbf9ef9528a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:01.304490 7266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f89f8faca5b4ec7918bdbf9ef9528a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:01.322593 7626 tablet_bootstrap.cc:492] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:01.324359 7626 tablet_bootstrap.cc:654] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:01.338743 7632 tablet_bootstrap.cc:492] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:01.340370 7632 tablet_bootstrap.cc:654] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:01.342674 7626 tablet_bootstrap.cc:492] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:01.342938 7626 ts_tablet_manager.cc:1367] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.002s
I20240501 20:10:01.344280 7626 raft_consensus.cc:348] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:01.344727 7626 raft_consensus.cc:374] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:01.344882 7626 raft_consensus.cc:729] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:01.349170 7626 consensus_queue.cc:260] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:01.351269 7626 ts_tablet_manager.cc:1398] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 20:10:01.353065 7710 tablet_bootstrap.cc:492] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:01.355273 7632 tablet_bootstrap.cc:492] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:01.355441 7632 ts_tablet_manager.cc:1367] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.003s
I20240501 20:10:01.356321 7632 raft_consensus.cc:348] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:01.356490 7632 raft_consensus.cc:374] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:01.356578 7632 raft_consensus.cc:729] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:01.356809 7632 consensus_queue.cc:260] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:01.357609 7710 tablet_bootstrap.cc:654] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:01.357916 7632 ts_tablet_manager.cc:1398] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:10:01.366351 7710 tablet_bootstrap.cc:492] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:01.366672 7710 ts_tablet_manager.cc:1367] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20240501 20:10:01.367612 7710 raft_consensus.cc:348] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:01.368002 7710 raft_consensus.cc:374] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:01.368158 7710 raft_consensus.cc:729] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:01.368510 7710 consensus_queue.cc:260] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:01.371922 7710 ts_tablet_manager.cc:1398] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 20:10:01.395224 4091 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240501 20:10:01.395480 4091 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240501 20:10:01.396802 7538 consensus_queue.cc:1035] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:01.400645 4091 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet e4b9e63c8c504df198d5774115a57098
I20240501 20:10:01.402019 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet e4b9e63c8c504df198d5774115a57098 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:01 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:01.406857 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet e4b9e63c8c504df198d5774115a57098 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:01 UTC) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:01.407208 7512 tablet_replica.cc:330] T e4b9e63c8c504df198d5774115a57098 P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:01.407353 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet e4b9e63c8c504df198d5774115a57098 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:01 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:01.407487 7512 raft_consensus.cc:2229] T e4b9e63c8c504df198d5774115a57098 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:01.407650 7519 tablet_replica.cc:330] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:01.407822 7512 raft_consensus.cc:2258] T e4b9e63c8c504df198d5774115a57098 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:01.407855 7519 raft_consensus.cc:2229] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:01.408208 7519 raft_consensus.cc:2258] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:01.408865 7512 ts_tablet_manager.cc:1875] T e4b9e63c8c504df198d5774115a57098 P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:01.409006 7519 ts_tablet_manager.cc:1875] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:01.411870 7651 consensus_queue.cc:1035] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:01.413499 7537 raft_consensus.cc:482] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:01.413661 7537 raft_consensus.cc:504] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:01.414497 7537 leader_election.cc:288] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:01.415521 4091 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 814749d88a674a278054b44a36a15770
I20240501 20:10:01.415521 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f89f8faca5b4ec7918bdbf9ef9528a6" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:01.415760 7512 ts_tablet_manager.cc:1888] T e4b9e63c8c504df198d5774115a57098 P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:01.415776 7286 raft_consensus.cc:2454] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:01.415880 7512 log.cc:1196] T e4b9e63c8c504df198d5774115a57098 P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/e4b9e63c8c504df198d5774115a57098
I20240501 20:10:01.416296 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet 814749d88a674a278054b44a36a15770 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:01 UTC) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:01.416368 7512 ts_tablet_manager.cc:1909] T e4b9e63c8c504df198d5774115a57098 P 10b47782558c41ee91f5e164a372049a: Deleting consensus metadata
I20240501 20:10:01.416317 6956 leader_election.cc:302] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:01.416625 7659 raft_consensus.cc:2781] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:01.416630 7641 tablet_replica.cc:330] T 814749d88a674a278054b44a36a15770 P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:01.416715 7659 raft_consensus.cc:482] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:01.416827 7659 raft_consensus.cc:3037] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:01.416872 7641 raft_consensus.cc:2229] T 814749d88a674a278054b44a36a15770 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:01.417138 7641 raft_consensus.cc:2258] T 814749d88a674a278054b44a36a15770 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:01.417915 7641 ts_tablet_manager.cc:1875] T 814749d88a674a278054b44a36a15770 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:01.418430 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet 814749d88a674a278054b44a36a15770 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:01 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:01.419211 4091 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 866614e2559a467ca9f746fa9cfd0f7d
I20240501 20:10:01.419448 7659 raft_consensus.cc:504] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:01.420550 4609 tablet_service.cc:1514] Processing DeleteTablet for tablet 814749d88a674a278054b44a36a15770 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:01 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:01.420095 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet e4b9e63c8c504df198d5774115a57098 (table rebalance_test_table_6 [id=1cba02e90ede4149af49f3f3763a48a9]) successfully deleted
I20240501 20:10:01.421556 4609 tablet_service.cc:1514] Processing DeleteTablet for tablet 866614e2559a467ca9f746fa9cfd0f7d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:01 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:01.422202 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f89f8faca5b4ec7918bdbf9ef9528a6" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:01.422488 7286 raft_consensus.cc:3037] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:01.422688 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet 866614e2559a467ca9f746fa9cfd0f7d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:01 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:01.424027 7519 ts_tablet_manager.cc:1888] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:01.424155 7519 log.cc:1196] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/e4b9e63c8c504df198d5774115a57098
I20240501 20:10:01.424602 7519 ts_tablet_manager.cc:1909] T e4b9e63c8c504df198d5774115a57098 P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:01.425163 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet 866614e2559a467ca9f746fa9cfd0f7d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:01 UTC) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:01.425631 7512 tablet_replica.cc:330] T 866614e2559a467ca9f746fa9cfd0f7d P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:01.426561 7659 leader_election.cc:288] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Requested vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:01.427490 7286 raft_consensus.cc:2454] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:01.427636 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet e4b9e63c8c504df198d5774115a57098 (table rebalance_test_table_6 [id=1cba02e90ede4149af49f3f3763a48a9]) successfully deleted
I20240501 20:10:01.428237 6956 leader_election.cc:302] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:01.429904 7659 raft_consensus.cc:2781] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:01.430078 7659 raft_consensus.cc:686] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Becoming Leader. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Running, Role: LEADER
I20240501 20:10:01.431052 7512 raft_consensus.cc:2229] T 866614e2559a467ca9f746fa9cfd0f7d P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:01.431315 7512 raft_consensus.cc:2258] T 866614e2559a467ca9f746fa9cfd0f7d P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:01.431289 7659 consensus_queue.cc:237] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:01.431588 7519 tablet_replica.cc:330] T 814749d88a674a278054b44a36a15770 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:01.431838 7519 raft_consensus.cc:2229] T 814749d88a674a278054b44a36a15770 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:01.432088 7519 raft_consensus.cc:2258] T 814749d88a674a278054b44a36a15770 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:01.432168 7512 ts_tablet_manager.cc:1875] T 866614e2559a467ca9f746fa9cfd0f7d P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:01.434159 7519 ts_tablet_manager.cc:1875] T 814749d88a674a278054b44a36a15770 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:01.436270 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f89f8faca5b4ec7918bdbf9ef9528a6" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:01.436373 7421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f89f8faca5b4ec7918bdbf9ef9528a6" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:01.436553 7420 raft_consensus.cc:2454] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:01.442991 7722 tablet_replica.cc:330] T e4b9e63c8c504df198d5774115a57098 P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:01.443336 7722 raft_consensus.cc:2229] T e4b9e63c8c504df198d5774115a57098 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:01.443749 7722 raft_consensus.cc:2258] T e4b9e63c8c504df198d5774115a57098 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:01.445040 7722 ts_tablet_manager.cc:1875] T e4b9e63c8c504df198d5774115a57098 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:01.451728 7519 ts_tablet_manager.cc:1888] T 814749d88a674a278054b44a36a15770 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:01.451717 7512 ts_tablet_manager.cc:1888] T 866614e2559a467ca9f746fa9cfd0f7d P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:01.451932 7512 log.cc:1196] T 866614e2559a467ca9f746fa9cfd0f7d P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/866614e2559a467ca9f746fa9cfd0f7d
I20240501 20:10:01.452576 7512 ts_tablet_manager.cc:1909] T 866614e2559a467ca9f746fa9cfd0f7d P 10b47782558c41ee91f5e164a372049a: Deleting consensus metadata
I20240501 20:10:01.454087 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet 866614e2559a467ca9f746fa9cfd0f7d (table rebalance_test_table_6 [id=1cba02e90ede4149af49f3f3763a48a9]) successfully deleted
I20240501 20:10:01.454324 7519 log.cc:1196] T 814749d88a674a278054b44a36a15770 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/814749d88a674a278054b44a36a15770
I20240501 20:10:01.454828 7519 ts_tablet_manager.cc:1909] T 814749d88a674a278054b44a36a15770 P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:01.457206 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet 814749d88a674a278054b44a36a15770 (table rebalance_test_table_6 [id=1cba02e90ede4149af49f3f3763a48a9]) successfully deleted
I20240501 20:10:01.458247 4087 catalog_manager.cc:5484] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69). New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:01.459846 7641 ts_tablet_manager.cc:1888] T 814749d88a674a278054b44a36a15770 P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:01.459995 7641 log.cc:1196] T 814749d88a674a278054b44a36a15770 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/814749d88a674a278054b44a36a15770
I20240501 20:10:01.460443 7641 ts_tablet_manager.cc:1909] T 814749d88a674a278054b44a36a15770 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting consensus metadata
I20240501 20:10:01.460479 7519 tablet_replica.cc:330] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:01.460902 7519 raft_consensus.cc:2229] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:01.461287 7519 raft_consensus.cc:2258] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:01.462443 7519 ts_tablet_manager.cc:1875] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:01.463312 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet 814749d88a674a278054b44a36a15770 (table rebalance_test_table_6 [id=1cba02e90ede4149af49f3f3763a48a9]) successfully deleted
I20240501 20:10:01.466198 7722 ts_tablet_manager.cc:1888] T e4b9e63c8c504df198d5774115a57098 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:01.466382 7722 log.cc:1196] T e4b9e63c8c504df198d5774115a57098 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/e4b9e63c8c504df198d5774115a57098
I20240501 20:10:01.466935 7722 ts_tablet_manager.cc:1909] T e4b9e63c8c504df198d5774115a57098 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:01.468014 7722 tablet_replica.cc:330] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:01.468115 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet e4b9e63c8c504df198d5774115a57098 (table rebalance_test_table_6 [id=1cba02e90ede4149af49f3f3763a48a9]) successfully deleted
I20240501 20:10:01.468261 7722 raft_consensus.cc:2229] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:01.468678 7722 raft_consensus.cc:2258] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:01.469861 7722 ts_tablet_manager.cc:1875] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:01.473603 7519 ts_tablet_manager.cc:1888] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:01.473763 7519 log.cc:1196] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/866614e2559a467ca9f746fa9cfd0f7d
I20240501 20:10:01.474298 7519 ts_tablet_manager.cc:1909] T 866614e2559a467ca9f746fa9cfd0f7d P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:01.480540 7722 ts_tablet_manager.cc:1888] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:01.481731 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet 866614e2559a467ca9f746fa9cfd0f7d (table rebalance_test_table_6 [id=1cba02e90ede4149af49f3f3763a48a9]) successfully deleted
I20240501 20:10:01.482861 7722 log.cc:1196] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/814749d88a674a278054b44a36a15770
I20240501 20:10:01.483458 7722 ts_tablet_manager.cc:1909] T 814749d88a674a278054b44a36a15770 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:01.484475 7722 tablet_replica.cc:330] T 866614e2559a467ca9f746fa9cfd0f7d P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:01.484752 7722 raft_consensus.cc:2229] T 866614e2559a467ca9f746fa9cfd0f7d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:01.485078 7722 raft_consensus.cc:2258] T 866614e2559a467ca9f746fa9cfd0f7d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:01.485260 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet 814749d88a674a278054b44a36a15770 (table rebalance_test_table_6 [id=1cba02e90ede4149af49f3f3763a48a9]) successfully deleted
I20240501 20:10:01.486008 7722 ts_tablet_manager.cc:1875] T 866614e2559a467ca9f746fa9cfd0f7d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:01.493103 7722 ts_tablet_manager.cc:1888] T 866614e2559a467ca9f746fa9cfd0f7d P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:01.493366 7722 log.cc:1196] T 866614e2559a467ca9f746fa9cfd0f7d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/866614e2559a467ca9f746fa9cfd0f7d
I20240501 20:10:01.493917 7722 ts_tablet_manager.cc:1909] T 866614e2559a467ca9f746fa9cfd0f7d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:01.495415 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet 866614e2559a467ca9f746fa9cfd0f7d (table rebalance_test_table_6 [id=1cba02e90ede4149af49f3f3763a48a9]) successfully deleted
I20240501 20:10:01.505885 7205 consensus_queue.cc:1035] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:10:01.519059 4347 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:01.537364 7401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:10:01.538733 4610 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:10:01.564378 4478 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 20:10:01.572580 7658 consensus_queue.cc:1035] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:01.580017 4084 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:01.591231 7421 raft_consensus.cc:3037] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:01.591276 7285 raft_consensus.cc:1261] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:01.591934 7537 consensus_queue.cc:1035] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:01.594487 7421 raft_consensus.cc:1261] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:01.596669 7537 consensus_queue.cc:1035] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:01.602996 4084 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:01.623031 7133 tablet_service.cc:1467] Processing CreateTablet for tablet 55da8e421b23413f96578a4af135b499 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=061e2b9b5d53488cbfdc25cfa8acdd76]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:01.623567 7133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55da8e421b23413f96578a4af135b499. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:01.624564 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 6f2576acd7d34ec1ad27ab04a863df10 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=061e2b9b5d53488cbfdc25cfa8acdd76]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:01.624699 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 55da8e421b23413f96578a4af135b499 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=061e2b9b5d53488cbfdc25cfa8acdd76]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:01.625046 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f2576acd7d34ec1ad27ab04a863df10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:01.625092 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55da8e421b23413f96578a4af135b499. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:01.626089 4347 tablet_service.cc:1467] Processing CreateTablet for tablet 6f2576acd7d34ec1ad27ab04a863df10 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=061e2b9b5d53488cbfdc25cfa8acdd76]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:01.626331 7002 tablet_service.cc:1467] Processing CreateTablet for tablet 4471aeb3f16f49d9bac12befd4cfd7e7 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=061e2b9b5d53488cbfdc25cfa8acdd76]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:01.626472 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f2576acd7d34ec1ad27ab04a863df10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:01.626870 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4471aeb3f16f49d9bac12befd4cfd7e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:01.630172 7400 tablet_service.cc:1467] Processing CreateTablet for tablet 4471aeb3f16f49d9bac12befd4cfd7e7 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=061e2b9b5d53488cbfdc25cfa8acdd76]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:01.630640 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4471aeb3f16f49d9bac12befd4cfd7e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:01.634097 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 4471aeb3f16f49d9bac12befd4cfd7e7 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=061e2b9b5d53488cbfdc25cfa8acdd76]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:01.634543 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4471aeb3f16f49d9bac12befd4cfd7e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:01.637470 7626 tablet_bootstrap.cc:492] T 4471aeb3f16f49d9bac12befd4cfd7e7 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:01.642947 7710 tablet_bootstrap.cc:492] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:01.645002 7710 tablet_bootstrap.cc:654] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:01.646211 7266 tablet_service.cc:1467] Processing CreateTablet for tablet 55da8e421b23413f96578a4af135b499 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=061e2b9b5d53488cbfdc25cfa8acdd76]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:01.646662 7266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55da8e421b23413f96578a4af135b499. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:01.647648 7626 tablet_bootstrap.cc:654] T 4471aeb3f16f49d9bac12befd4cfd7e7 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:01.651166 7267 tablet_service.cc:1467] Processing CreateTablet for tablet 6f2576acd7d34ec1ad27ab04a863df10 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=061e2b9b5d53488cbfdc25cfa8acdd76]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:01.651708 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f2576acd7d34ec1ad27ab04a863df10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:01.656812 7537 consensus_queue.cc:1035] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:01.659494 7747 tablet_bootstrap.cc:492] T 6f2576acd7d34ec1ad27ab04a863df10 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:01.663389 7748 tablet_bootstrap.cc:492] T 55da8e421b23413f96578a4af135b499 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:01.673053 7724 mvcc.cc:205] Tried to move back new op lower bound from 7022977849707732992 to 7022977849069740032. Current Snapshot: MvccSnapshot[applied={T|T < 7022977849707732992}]
I20240501 20:10:01.682360 7632 tablet_bootstrap.cc:492] T 55da8e421b23413f96578a4af135b499 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:01.684325 7632 tablet_bootstrap.cc:654] T 55da8e421b23413f96578a4af135b499 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:01.685102 7626 tablet_bootstrap.cc:492] T 4471aeb3f16f49d9bac12befd4cfd7e7 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:01.685305 7626 ts_tablet_manager.cc:1367] T 4471aeb3f16f49d9bac12befd4cfd7e7 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.005s
I20240501 20:10:01.686344 7626 raft_consensus.cc:348] T 4471aeb3f16f49d9bac12befd4cfd7e7 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:01.686677 7626 raft_consensus.cc:374] T 4471aeb3f16f49d9bac12befd4cfd7e7 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:01.686764 7626 raft_consensus.cc:729] T 4471aeb3f16f49d9bac12befd4cfd7e7 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:01.686959 4084 catalog_manager.cc:5576] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237) does not have the latest schema for tablet 7f89f8faca5b4ec7918bdbf9ef9528a6 (table rebalancer_dynamic_table [id=da2afc94765b4977ad891669f4f8ede2]). Expected version 1 got 0
I20240501 20:10:01.687072 7626 consensus_queue.cc:260] T 4471aeb3f16f49d9bac12befd4cfd7e7 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:01.683422 7750 tablet.cc:1722] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:01.688628 7710 tablet_bootstrap.cc:492] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:01.688870 7710 ts_tablet_manager.cc:1367] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.046s user 0.008s sys 0.004s
I20240501 20:10:01.689698 7750 tablet.cc:1620] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:01.690085 7744 tablet_bootstrap.cc:492] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:01.690207 7710 raft_consensus.cc:348] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:01.690531 7710 raft_consensus.cc:374] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:01.690658 7710 raft_consensus.cc:729] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:01.691051 7710 consensus_queue.cc:260] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:01.692302 7744 tablet_bootstrap.cc:654] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:01.697913 7626 ts_tablet_manager.cc:1398] T 4471aeb3f16f49d9bac12befd4cfd7e7 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240501 20:10:01.697877 7659 consensus_queue.cc:1035] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:10:01.698637 7710 ts_tablet_manager.cc:1398] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.010s user 0.000s sys 0.003s
I20240501 20:10:01.698989 7710 tablet_bootstrap.cc:492] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:01.700809 7710 tablet_bootstrap.cc:654] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:01.701931 7741 tablet_bootstrap.cc:492] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:01.696611 7751 tablet.cc:1722] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:01.709707 7751 tablet.cc:1620] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:01.712474 4087 catalog_manager.cc:5576] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447) does not have the latest schema for tablet 7f89f8faca5b4ec7918bdbf9ef9528a6 (table rebalancer_dynamic_table [id=da2afc94765b4977ad891669f4f8ede2]). Expected version 1 got 0
I20240501 20:10:01.713279 4091 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=da2afc94765b4977ad891669f4f8ede2] alter complete (version 1)
I20240501 20:10:01.721305 7741 tablet_bootstrap.cc:654] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:01.741462 7747 tablet_bootstrap.cc:654] T 6f2576acd7d34ec1ad27ab04a863df10 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:01.751076 7748 tablet_bootstrap.cc:654] T 55da8e421b23413f96578a4af135b499 P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:01.755649 7741 tablet_bootstrap.cc:492] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:01.756021 7741 ts_tablet_manager.cc:1367] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.054s user 0.000s sys 0.007s
I20240501 20:10:01.762722 7741 raft_consensus.cc:348] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:01.765796 7632 tablet_bootstrap.cc:492] T 55da8e421b23413f96578a4af135b499 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:01.766158 7632 ts_tablet_manager.cc:1367] T 55da8e421b23413f96578a4af135b499 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.084s user 0.006s sys 0.006s
I20240501 20:10:01.767601 7632 raft_consensus.cc:348] T 55da8e421b23413f96578a4af135b499 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:01.770365 7632 raft_consensus.cc:374] T 55da8e421b23413f96578a4af135b499 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:01.770526 7632 raft_consensus.cc:729] T 55da8e421b23413f96578a4af135b499 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:01.771678 7710 tablet_bootstrap.cc:492] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:01.771896 7710 ts_tablet_manager.cc:1367] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.073s user 0.004s sys 0.002s
I20240501 20:10:01.772665 7710 raft_consensus.cc:348] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:01.773046 7710 raft_consensus.cc:374] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:01.773203 7710 raft_consensus.cc:729] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:01.773588 7710 consensus_queue.cc:260] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:01.774915 7710 ts_tablet_manager.cc:1398] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:10:01.776043 4090 catalog_manager.cc:5576] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447) does not have the latest schema for tablet 7f89f8faca5b4ec7918bdbf9ef9528a6 (table rebalancer_dynamic_table [id=da2afc94765b4977ad891669f4f8ede2]). Expected version 1 got 0
I20240501 20:10:01.773254 7632 consensus_queue.cc:260] T 55da8e421b23413f96578a4af135b499 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:01.782369 7133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:01.783075 7741 raft_consensus.cc:374] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:01.783342 7741 raft_consensus.cc:729] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:01.783735 7741 consensus_queue.cc:260] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:01.785890 7267 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:01.787673 7632 ts_tablet_manager.cc:1398] T 55da8e421b23413f96578a4af135b499 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240501 20:10:01.787776 7741 ts_tablet_manager.cc:1398] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.026s user 0.003s sys 0.000s
I20240501 20:10:01.790299 7632 tablet_bootstrap.cc:492] T 6f2576acd7d34ec1ad27ab04a863df10 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:01.796519 7632 tablet_bootstrap.cc:654] T 6f2576acd7d34ec1ad27ab04a863df10 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:01.799080 7516 raft_consensus.cc:482] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:01.799239 7516 raft_consensus.cc:504] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:01.800156 7516 leader_election.cc:288] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:01.800866 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f2576acd7d34ec1ad27ab04a863df10" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
W20240501 20:10:01.801781 4564 leader_election.cc:341] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:01.803591 7747 tablet_bootstrap.cc:492] T 6f2576acd7d34ec1ad27ab04a863df10 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:10:01.773378 7762 tablet.cc:1722] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:01.819528 7762 tablet.cc:1620] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:01.819875 7747 ts_tablet_manager.cc:1367] T 6f2576acd7d34ec1ad27ab04a863df10 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.160s user 0.005s sys 0.000s
I20240501 20:10:01.823060 7744 tablet_bootstrap.cc:492] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:01.823315 7744 ts_tablet_manager.cc:1367] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.133s user 0.016s sys 0.013s
I20240501 20:10:01.824232 7744 raft_consensus.cc:348] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:01.824524 7744 raft_consensus.cc:374] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:01.824678 7744 raft_consensus.cc:729] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:01.825016 7744 consensus_queue.cc:260] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:01.829957 7744 ts_tablet_manager.cc:1398] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:10:01.832185 4087 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:01.832975 7747 raft_consensus.cc:348] T 6f2576acd7d34ec1ad27ab04a863df10 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:01.833222 7747 raft_consensus.cc:374] T 6f2576acd7d34ec1ad27ab04a863df10 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:01.833315 7747 raft_consensus.cc:729] T 6f2576acd7d34ec1ad27ab04a863df10 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:10:01.833578 7747 consensus_queue.cc:260] T 6f2576acd7d34ec1ad27ab04a863df10 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:01.843159 7760 raft_consensus.cc:482] T 4471aeb3f16f49d9bac12befd4cfd7e7 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:01.843386 7760 raft_consensus.cc:504] T 4471aeb3f16f49d9bac12befd4cfd7e7 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:01.844408 7760 leader_election.cc:288] T 4471aeb3f16f49d9bac12befd4cfd7e7 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
W20240501 20:10:01.844908 4677 tablet.cc:2369] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:10:01.844976 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4471aeb3f16f49d9bac12befd4cfd7e7" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:01.845199 7420 raft_consensus.cc:2454] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:01.845641 6958 leader_election.cc:302] T 4471aeb3f16f49d9bac12befd4cfd7e7 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:01.846055 7760 raft_consensus.cc:2781] T 4471aeb3f16f49d9bac12befd4cfd7e7 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:01.846165 7760 raft_consensus.cc:482] T 4471aeb3f16f49d9bac12befd4cfd7e7 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:01.846242 7760 raft_consensus.cc:3037] T 4471aeb3f16f49d9bac12befd4cfd7e7 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:01.849496 7760 raft_consensus.cc:504] T 4471aeb3f16f49d9bac12befd4cfd7e7 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:01.850839 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4471aeb3f16f49d9bac12befd4cfd7e7" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:01.851102 7420 raft_consensus.cc:3037] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:01.854161 7420 raft_consensus.cc:2454] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:01.855058 6958 leader_election.cc:302] T 4471aeb3f16f49d9bac12befd4cfd7e7 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:01.855713 7758 raft_consensus.cc:2781] T 4471aeb3f16f49d9bac12befd4cfd7e7 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:01.856060 7758 raft_consensus.cc:686] T 4471aeb3f16f49d9bac12befd4cfd7e7 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Becoming Leader. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Running, Role: LEADER
I20240501 20:10:01.856693 7758 consensus_queue.cc:237] T 4471aeb3f16f49d9bac12befd4cfd7e7 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:01.857916 7760 leader_election.cc:288] T 4471aeb3f16f49d9bac12befd4cfd7e7 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Requested vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:01.866796 7747 ts_tablet_manager.cc:1398] T 6f2576acd7d34ec1ad27ab04a863df10 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240501 20:10:01.870354 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4471aeb3f16f49d9bac12befd4cfd7e7" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:01.870648 4233 raft_consensus.cc:2454] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:01.876269 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4471aeb3f16f49d9bac12befd4cfd7e7" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:10:01.876526 4233 raft_consensus.cc:3037] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:01.876863 7751 tablet.cc:1722] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:01.877362 7751 tablet.cc:1620] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:01.879514 4233 raft_consensus.cc:2454] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:01.883702 7748 tablet_bootstrap.cc:492] T 55da8e421b23413f96578a4af135b499 P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:01.883868 7748 ts_tablet_manager.cc:1367] T 55da8e421b23413f96578a4af135b499 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.221s user 0.002s sys 0.004s
I20240501 20:10:01.884780 7748 raft_consensus.cc:348] T 55da8e421b23413f96578a4af135b499 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:01.885069 7748 raft_consensus.cc:374] T 55da8e421b23413f96578a4af135b499 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:01.885787 7748 raft_consensus.cc:729] T 55da8e421b23413f96578a4af135b499 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:01.886075 7748 consensus_queue.cc:260] T 55da8e421b23413f96578a4af135b499 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:01.888178 4090 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=da2afc94765b4977ad891669f4f8ede2] alter complete (version 2)
I20240501 20:10:01.890177 7750 tablet.cc:1722] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:01.891129 7750 tablet.cc:1620] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:01.897189 4087 catalog_manager.cc:5576] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447) does not have the latest schema for tablet 7f89f8faca5b4ec7918bdbf9ef9528a6 (table rebalancer_dynamic_table [id=da2afc94765b4977ad891669f4f8ede2]). Expected version 2 got 1
I20240501 20:10:01.901820 7774 tablet.cc:1722] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:01.902999 7774 tablet.cc:1620] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:01.908368 7495 raft_consensus.cc:482] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:01.908543 7495 raft_consensus.cc:504] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:01.909307 4215 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:01.909303 7495 leader_election.cc:288] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:01.910372 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55da8e421b23413f96578a4af135b499" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:01.910660 7284 raft_consensus.cc:2454] T 55da8e421b23413f96578a4af135b499 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:10:01.911252 7354 leader_election.cc:302] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:01.911613 7154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55da8e421b23413f96578a4af135b499" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:01.911924 7154 raft_consensus.cc:2454] T 55da8e421b23413f96578a4af135b499 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:10:01.912319 7495 raft_consensus.cc:2781] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:01.912441 7495 raft_consensus.cc:482] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:01.912536 7495 raft_consensus.cc:3037] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:01.912607 7748 ts_tablet_manager.cc:1398] T 55da8e421b23413f96578a4af135b499 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240501 20:10:01.924090 4090 catalog_manager.cc:5484] T 4471aeb3f16f49d9bac12befd4cfd7e7 P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69). New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:01.930292 4087 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240501 20:10:01.930599 4087 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240501 20:10:01.940062 4087 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 85d8684f2bed453c9f0535ce2ac1c6a2
I20240501 20:10:01.940747 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet 85d8684f2bed453c9f0535ce2ac1c6a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:01 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:01.941037 7519 tablet_replica.cc:330] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:01.941004 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet 85d8684f2bed453c9f0535ce2ac1c6a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:01 UTC) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:01.941272 7519 raft_consensus.cc:2229] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:01.941301 7641 tablet_replica.cc:330] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:01.941450 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet 85d8684f2bed453c9f0535ce2ac1c6a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:01 UTC) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:01.941512 7641 raft_consensus.cc:2229] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:01.941606 7519 raft_consensus.cc:2258] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:01.941751 7512 tablet_replica.cc:330] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:01.941795 7641 raft_consensus.cc:2258] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:01.941928 7512 raft_consensus.cc:2229] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:01.942140 7512 raft_consensus.cc:2258] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:01.942406 7519 ts_tablet_manager.cc:1875] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:01.942456 7641 ts_tablet_manager.cc:1875] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:01.944717 7512 ts_tablet_manager.cc:1875] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:01.949122 7641 ts_tablet_manager.cc:1888] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:01.949357 7641 log.cc:1196] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/85d8684f2bed453c9f0535ce2ac1c6a2
I20240501 20:10:01.949914 7641 ts_tablet_manager.cc:1909] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting consensus metadata
I20240501 20:10:01.949980 7519 ts_tablet_manager.cc:1888] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:01.950120 7519 log.cc:1196] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/85d8684f2bed453c9f0535ce2ac1c6a2
I20240501 20:10:01.950512 7519 ts_tablet_manager.cc:1909] T 85d8684f2bed453c9f0535ce2ac1c6a2 P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:01.951397 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet 85d8684f2bed453c9f0535ce2ac1c6a2 (table rebalance_test_table_8 [id=52695e0315884c0aa878488af56a1006]) successfully deleted
I20240501 20:10:01.952634 4087 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet ff61519287b64c35a573cfca87fc4dc8
I20240501 20:10:01.921137 7495 raft_consensus.cc:504] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:01.954223 7495 leader_election.cc:288] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Requested vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:01.955255 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet ff61519287b64c35a573cfca87fc4dc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:01 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:01.955647 7154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55da8e421b23413f96578a4af135b499" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:01.955888 7154 raft_consensus.cc:3037] T 55da8e421b23413f96578a4af135b499 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:01.956058 4087 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet e7d2e3f5d38e423dbc905aeabe3dbef8
I20240501 20:10:01.956185 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55da8e421b23413f96578a4af135b499" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:01.956409 7284 raft_consensus.cc:3037] T 55da8e421b23413f96578a4af135b499 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:01.957357 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet e7d2e3f5d38e423dbc905aeabe3dbef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:01 UTC) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:01.957749 7641 tablet_replica.cc:330] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:01.958077 7641 raft_consensus.cc:2229] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:01.958495 7641 raft_consensus.cc:2258] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:01.958804 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet ff61519287b64c35a573cfca87fc4dc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:01 UTC) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:01.959300 7641 ts_tablet_manager.cc:1875] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:01.959895 7154 raft_consensus.cc:2454] T 55da8e421b23413f96578a4af135b499 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:01.960445 7353 leader_election.cc:302] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:01.960928 7495 raft_consensus.cc:2781] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:01.960958 7512 ts_tablet_manager.cc:1888] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:01.961095 7512 log.cc:1196] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/85d8684f2bed453c9f0535ce2ac1c6a2
I20240501 20:10:01.961568 7512 ts_tablet_manager.cc:1909] T 85d8684f2bed453c9f0535ce2ac1c6a2 P 10b47782558c41ee91f5e164a372049a: Deleting consensus metadata
I20240501 20:10:01.962719 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet 85d8684f2bed453c9f0535ce2ac1c6a2 (table rebalance_test_table_8 [id=52695e0315884c0aa878488af56a1006]) successfully deleted
I20240501 20:10:01.963352 7284 raft_consensus.cc:2454] T 55da8e421b23413f96578a4af135b499 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:01.963470 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet e7d2e3f5d38e423dbc905aeabe3dbef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:01 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:01.968106 7641 ts_tablet_manager.cc:1888] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:01.971602 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet 85d8684f2bed453c9f0535ce2ac1c6a2 (table rebalance_test_table_8 [id=52695e0315884c0aa878488af56a1006]) successfully deleted
I20240501 20:10:01.972602 4609 tablet_service.cc:1514] Processing DeleteTablet for tablet e7d2e3f5d38e423dbc905aeabe3dbef8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:01 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:01.972911 7722 tablet_replica.cc:330] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:01.973528 7722 raft_consensus.cc:2229] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:01.973771 7722 raft_consensus.cc:2258] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:01.974748 7722 ts_tablet_manager.cc:1875] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:01.975335 7641 log.cc:1196] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/e7d2e3f5d38e423dbc905aeabe3dbef8
I20240501 20:10:01.975809 7641 ts_tablet_manager.cc:1909] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting consensus metadata
I20240501 20:10:01.975775 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet ff61519287b64c35a573cfca87fc4dc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:01 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:01.976763 7002 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:01.976902 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet e7d2e3f5d38e423dbc905aeabe3dbef8 (table rebalance_test_table_8 [id=52695e0315884c0aa878488af56a1006]) successfully deleted
I20240501 20:10:01.983147 7495 raft_consensus.cc:686] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Becoming Leader. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Running, Role: LEADER
I20240501 20:10:01.983597 7495 consensus_queue.cc:237] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:01.992816 4087 catalog_manager.cc:5484] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72). New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:01.995029 4090 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:01.997251 7722 ts_tablet_manager.cc:1888] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:01.997395 7722 log.cc:1196] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/e7d2e3f5d38e423dbc905aeabe3dbef8
I20240501 20:10:01.997769 7722 ts_tablet_manager.cc:1909] T e7d2e3f5d38e423dbc905aeabe3dbef8 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:01.998828 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet e7d2e3f5d38e423dbc905aeabe3dbef8 (table rebalance_test_table_8 [id=52695e0315884c0aa878488af56a1006]) successfully deleted
I20240501 20:10:02.003484 7722 tablet_replica.cc:330] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:02.003701 7722 raft_consensus.cc:2229] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:02.003990 7722 raft_consensus.cc:2258] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:02.004905 7722 ts_tablet_manager.cc:1875] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:02.006430 7750 tablet.cc:1722] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:02.007261 7632 tablet_bootstrap.cc:492] T 6f2576acd7d34ec1ad27ab04a863df10 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:02.007494 7632 ts_tablet_manager.cc:1367] T 6f2576acd7d34ec1ad27ab04a863df10 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.217s user 0.011s sys 0.039s
I20240501 20:10:02.008878 7519 tablet_replica.cc:330] T ff61519287b64c35a573cfca87fc4dc8 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:02.009776 7519 raft_consensus.cc:2229] T ff61519287b64c35a573cfca87fc4dc8 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:02.012324 7722 ts_tablet_manager.cc:1888] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:02.012514 7722 log.cc:1196] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/ff61519287b64c35a573cfca87fc4dc8
I20240501 20:10:02.012979 7774 tablet.cc:1722] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:02.010238 7519 raft_consensus.cc:2258] T ff61519287b64c35a573cfca87fc4dc8 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:02.013554 7751 tablet.cc:1722] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:02.015085 7519 ts_tablet_manager.cc:1875] T ff61519287b64c35a573cfca87fc4dc8 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:02.015070 7632 raft_consensus.cc:348] T 6f2576acd7d34ec1ad27ab04a863df10 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:02.015301 7632 raft_consensus.cc:374] T 6f2576acd7d34ec1ad27ab04a863df10 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:02.015406 7632 raft_consensus.cc:729] T 6f2576acd7d34ec1ad27ab04a863df10 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:02.015446 7512 tablet_replica.cc:330] T ff61519287b64c35a573cfca87fc4dc8 P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:02.015753 7512 raft_consensus.cc:2229] T ff61519287b64c35a573cfca87fc4dc8 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:02.015728 7632 consensus_queue.cc:260] T 6f2576acd7d34ec1ad27ab04a863df10 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:02.016858 7632 ts_tablet_manager.cc:1398] T 6f2576acd7d34ec1ad27ab04a863df10 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 20:10:02.020403 7512 raft_consensus.cc:2258] T ff61519287b64c35a573cfca87fc4dc8 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:02.020992 7722 ts_tablet_manager.cc:1909] T ff61519287b64c35a573cfca87fc4dc8 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:02.021842 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f2576acd7d34ec1ad27ab04a863df10" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:02.022166 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet ff61519287b64c35a573cfca87fc4dc8 (table rebalance_test_table_8 [id=52695e0315884c0aa878488af56a1006]) successfully deleted
I20240501 20:10:02.022238 7284 raft_consensus.cc:2454] T 6f2576acd7d34ec1ad27ab04a863df10 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:10:02.022831 4563 leader_election.cc:302] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, bceeae1c79964acabdae8ff771bf0c97; no voters: 10b47782558c41ee91f5e164a372049a
I20240501 20:10:02.023182 7512 ts_tablet_manager.cc:1875] T ff61519287b64c35a573cfca87fc4dc8 P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:02.023298 7516 raft_consensus.cc:2781] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:02.023402 7516 raft_consensus.cc:482] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:02.023478 7516 raft_consensus.cc:3037] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:02.026151 7516 raft_consensus.cc:504] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:02.029055 4087 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table_new_name [id=da2afc94765b4977ad891669f4f8ede2] alter complete (version 3)
I20240501 20:10:02.031339 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f2576acd7d34ec1ad27ab04a863df10" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:02.031579 7284 raft_consensus.cc:3037] T 6f2576acd7d34ec1ad27ab04a863df10 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:02.034875 7516 leader_election.cc:288] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Requested vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:02.035387 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f2576acd7d34ec1ad27ab04a863df10" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:10:02.035629 4365 raft_consensus.cc:3037] T 6f2576acd7d34ec1ad27ab04a863df10 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:02.036381 7284 raft_consensus.cc:2454] T 6f2576acd7d34ec1ad27ab04a863df10 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:10:02.036809 4563 leader_election.cc:302] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:02.037938 7516 raft_consensus.cc:2781] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:02.038067 7516 raft_consensus.cc:686] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Becoming Leader. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Running, Role: LEADER
I20240501 20:10:02.038316 7516 consensus_queue.cc:237] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:02.038514 4365 raft_consensus.cc:2454] T 6f2576acd7d34ec1ad27ab04a863df10 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:10:02.047117 7512 ts_tablet_manager.cc:1888] T ff61519287b64c35a573cfca87fc4dc8 P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:02.047312 7512 log.cc:1196] T ff61519287b64c35a573cfca87fc4dc8 P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/ff61519287b64c35a573cfca87fc4dc8
I20240501 20:10:02.047852 7512 ts_tablet_manager.cc:1909] T ff61519287b64c35a573cfca87fc4dc8 P 10b47782558c41ee91f5e164a372049a: Deleting consensus metadata
I20240501 20:10:02.048692 7519 ts_tablet_manager.cc:1888] T ff61519287b64c35a573cfca87fc4dc8 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:02.049067 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet ff61519287b64c35a573cfca87fc4dc8 (table rebalance_test_table_8 [id=52695e0315884c0aa878488af56a1006]) successfully deleted
I20240501 20:10:02.050781 7519 log.cc:1196] T ff61519287b64c35a573cfca87fc4dc8 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/ff61519287b64c35a573cfca87fc4dc8
I20240501 20:10:02.051396 7519 ts_tablet_manager.cc:1909] T ff61519287b64c35a573cfca87fc4dc8 P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:02.051807 4090 catalog_manager.cc:5484] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68). New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:02.052590 7519 tablet_replica.cc:330] T e7d2e3f5d38e423dbc905aeabe3dbef8 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:02.053022 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet ff61519287b64c35a573cfca87fc4dc8 (table rebalance_test_table_8 [id=52695e0315884c0aa878488af56a1006]) successfully deleted
I20240501 20:10:02.053831 7519 raft_consensus.cc:2229] T e7d2e3f5d38e423dbc905aeabe3dbef8 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:02.055122 7519 raft_consensus.cc:2258] T e7d2e3f5d38e423dbc905aeabe3dbef8 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:02.055914 7519 ts_tablet_manager.cc:1875] T e7d2e3f5d38e423dbc905aeabe3dbef8 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:02.062026 7519 ts_tablet_manager.cc:1888] T e7d2e3f5d38e423dbc905aeabe3dbef8 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:02.062183 7519 log.cc:1196] T e7d2e3f5d38e423dbc905aeabe3dbef8 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/e7d2e3f5d38e423dbc905aeabe3dbef8
I20240501 20:10:02.063088 7519 ts_tablet_manager.cc:1909] T e7d2e3f5d38e423dbc905aeabe3dbef8 P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:02.064152 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet e7d2e3f5d38e423dbc905aeabe3dbef8 (table rebalance_test_table_8 [id=52695e0315884c0aa878488af56a1006]) successfully deleted
I20240501 20:10:02.132465 4090 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:02.132704 4090 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:02.136502 4090 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 7f89f8faca5b4ec7918bdbf9ef9528a6
I20240501 20:10:02.137681 7002 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f89f8faca5b4ec7918bdbf9ef9528a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:02 UTC) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:02.138540 7791 tablet_replica.cc:330] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d: stopping tablet replica
I20240501 20:10:02.138531 7400 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f89f8faca5b4ec7918bdbf9ef9528a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:02 UTC) from {username='slave'} at 127.0.0.1:40182
I20240501 20:10:02.138878 7267 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f89f8faca5b4ec7918bdbf9ef9528a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:02 UTC) from {username='slave'} at 127.0.0.1:54166
I20240501 20:10:02.139178 7791 raft_consensus.cc:2229] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:02.140132 7791 raft_consensus.cc:2258] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:02.142174 7791 ts_tablet_manager.cc:1875] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:02.149358 7793 tablet_replica.cc:330] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:02.149781 7793 raft_consensus.cc:2229] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:02.150125 7793 raft_consensus.cc:2258] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:02.151093 7793 ts_tablet_manager.cc:1875] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:02.151494 7791 ts_tablet_manager.cc:1888] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:02.151736 7791 log.cc:1196] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/wals/7f89f8faca5b4ec7918bdbf9ef9528a6
I20240501 20:10:02.152529 7791 ts_tablet_manager.cc:1909] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting consensus metadata
I20240501 20:10:02.154942 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet 7f89f8faca5b4ec7918bdbf9ef9528a6 (table rebalancer_dynamic_table_new_name [id=da2afc94765b4977ad891669f4f8ede2]) successfully deleted
I20240501 20:10:02.159482 7793 ts_tablet_manager.cc:1888] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:02.159610 7793 log.cc:1196] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/7f89f8faca5b4ec7918bdbf9ef9528a6
I20240501 20:10:02.160024 7793 ts_tablet_manager.cc:1909] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P bceeae1c79964acabdae8ff771bf0c97: Deleting consensus metadata
I20240501 20:10:02.160964 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet 7f89f8faca5b4ec7918bdbf9ef9528a6 (table rebalancer_dynamic_table_new_name [id=da2afc94765b4977ad891669f4f8ede2]) successfully deleted
I20240501 20:10:02.166999 7792 tablet_replica.cc:330] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P b9957d4086bc4a76b18ed9265a5739d5: stopping tablet replica
I20240501 20:10:02.167229 7792 raft_consensus.cc:2229] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:02.167445 7792 raft_consensus.cc:2258] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:02.168236 7792 ts_tablet_manager.cc:1875] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P b9957d4086bc4a76b18ed9265a5739d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:02.174503 7792 ts_tablet_manager.cc:1888] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P b9957d4086bc4a76b18ed9265a5739d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:02.174789 7792 log.cc:1196] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P b9957d4086bc4a76b18ed9265a5739d5: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/wals/7f89f8faca5b4ec7918bdbf9ef9528a6
I20240501 20:10:02.175343 7792 ts_tablet_manager.cc:1909] T 7f89f8faca5b4ec7918bdbf9ef9528a6 P b9957d4086bc4a76b18ed9265a5739d5: Deleting consensus metadata
I20240501 20:10:02.176695 4079 catalog_manager.cc:4830] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): tablet 7f89f8faca5b4ec7918bdbf9ef9528a6 (table rebalancer_dynamic_table_new_name [id=da2afc94765b4977ad891669f4f8ede2]) successfully deleted
I20240501 20:10:02.238837 4090 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:02.247913 7002 tablet_service.cc:1467] Processing CreateTablet for tablet 91ff8be30dc04f2fa844e8e32e0c46f4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8dc02548b7cf43e782b8568293e87f1e]), partition=
I20240501 20:10:02.248315 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91ff8be30dc04f2fa844e8e32e0c46f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:02.248452 7133 tablet_service.cc:1467] Processing CreateTablet for tablet 91ff8be30dc04f2fa844e8e32e0c46f4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8dc02548b7cf43e782b8568293e87f1e]), partition=
I20240501 20:10:02.248939 7133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91ff8be30dc04f2fa844e8e32e0c46f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:02.254065 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 91ff8be30dc04f2fa844e8e32e0c46f4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8dc02548b7cf43e782b8568293e87f1e]), partition=
I20240501 20:10:02.254491 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91ff8be30dc04f2fa844e8e32e0c46f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:02.261195 7760 consensus_queue.cc:1035] T 4471aeb3f16f49d9bac12befd4cfd7e7 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:02.263022 7744 tablet_bootstrap.cc:492] T 91ff8be30dc04f2fa844e8e32e0c46f4 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:02.267468 7748 tablet_bootstrap.cc:492] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:02.269779 7748 tablet_bootstrap.cc:654] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:02.274732 7744 tablet_bootstrap.cc:654] T 91ff8be30dc04f2fa844e8e32e0c46f4 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:02.277841 7808 tablet_bootstrap.cc:492] T 91ff8be30dc04f2fa844e8e32e0c46f4 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:02.280086 7808 tablet_bootstrap.cc:654] T 91ff8be30dc04f2fa844e8e32e0c46f4 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:02.280965 7748 tablet_bootstrap.cc:492] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:02.281231 7748 ts_tablet_manager.cc:1367] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.006s
I20240501 20:10:02.282035 7748 raft_consensus.cc:348] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:02.282351 7748 raft_consensus.cc:374] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:02.282466 7748 raft_consensus.cc:729] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:02.282958 7748 consensus_queue.cc:260] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:02.286656 7748 ts_tablet_manager.cc:1398] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:10:02.293839 7659 consensus_queue.cc:1035] T 4471aeb3f16f49d9bac12befd4cfd7e7 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:02.294072 7744 tablet_bootstrap.cc:492] T 91ff8be30dc04f2fa844e8e32e0c46f4 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:02.294237 7744 ts_tablet_manager.cc:1367] T 91ff8be30dc04f2fa844e8e32e0c46f4 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.008s
I20240501 20:10:02.295097 7744 raft_consensus.cc:348] T 91ff8be30dc04f2fa844e8e32e0c46f4 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:02.295846 7744 raft_consensus.cc:374] T 91ff8be30dc04f2fa844e8e32e0c46f4 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:02.295938 7744 raft_consensus.cc:729] T 91ff8be30dc04f2fa844e8e32e0c46f4 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:02.296241 7744 consensus_queue.cc:260] T 91ff8be30dc04f2fa844e8e32e0c46f4 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:02.297168 7744 ts_tablet_manager.cc:1398] T 91ff8be30dc04f2fa844e8e32e0c46f4 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:10:02.301713 7808 tablet_bootstrap.cc:492] T 91ff8be30dc04f2fa844e8e32e0c46f4 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:02.301961 7808 ts_tablet_manager.cc:1367] T 91ff8be30dc04f2fa844e8e32e0c46f4 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.004s
I20240501 20:10:02.305356 7808 raft_consensus.cc:348] T 91ff8be30dc04f2fa844e8e32e0c46f4 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:02.305696 7808 raft_consensus.cc:374] T 91ff8be30dc04f2fa844e8e32e0c46f4 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:02.305900 7808 raft_consensus.cc:729] T 91ff8be30dc04f2fa844e8e32e0c46f4 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:02.306264 7808 consensus_queue.cc:260] T 91ff8be30dc04f2fa844e8e32e0c46f4 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:02.307910 7808 ts_tablet_manager.cc:1398] T 91ff8be30dc04f2fa844e8e32e0c46f4 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240501 20:10:02.462211 4610 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:10:02.470573 4347 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:02.471094 4092 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240501 20:10:02.471359 4092 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240501 20:10:02.480260 4092 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 0b14eb56de3940de9fbf03d31b860005
I20240501 20:10:02.482345 4092 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 8afb9c6ed6f946eeb7a015a0f6f0c658
I20240501 20:10:02.482832 7133 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:02.483206 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b14eb56de3940de9fbf03d31b860005 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:02 UTC) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:02.483197 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b14eb56de3940de9fbf03d31b860005 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:02 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:02.483503 7512 tablet_replica.cc:330] T 0b14eb56de3940de9fbf03d31b860005 P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:02.483734 7512 raft_consensus.cc:2229] T 0b14eb56de3940de9fbf03d31b860005 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:02.484015 7512 raft_consensus.cc:2258] T 0b14eb56de3940de9fbf03d31b860005 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:02.484817 7512 ts_tablet_manager.cc:1875] T 0b14eb56de3940de9fbf03d31b860005 P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:02.485502 7267 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:02.486827 7722 tablet_replica.cc:330] T 0b14eb56de3940de9fbf03d31b860005 P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:02.487068 7722 raft_consensus.cc:2229] T 0b14eb56de3940de9fbf03d31b860005 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:02.487321 7722 raft_consensus.cc:2258] T 0b14eb56de3940de9fbf03d31b860005 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:02.487443 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet 8afb9c6ed6f946eeb7a015a0f6f0c658 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:02 UTC) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:02.488103 7722 ts_tablet_manager.cc:1875] T 0b14eb56de3940de9fbf03d31b860005 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:02.490753 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet 8afb9c6ed6f946eeb7a015a0f6f0c658 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:02 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:02.491689 4092 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 11c38f7be4804540806d743e1fa47b48
I20240501 20:10:02.493117 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet 11c38f7be4804540806d743e1fa47b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:02 UTC) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:02.493134 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet 11c38f7be4804540806d743e1fa47b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:02 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:02.494720 7551 raft_consensus.cc:482] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:02.494904 7551 raft_consensus.cc:504] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:02.495692 7551 leader_election.cc:288] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:02.496623 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ff8be30dc04f2fa844e8e32e0c46f4" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:02.496867 7022 raft_consensus.cc:2454] T 91ff8be30dc04f2fa844e8e32e0c46f4 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:02.496834 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ff8be30dc04f2fa844e8e32e0c46f4" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:02.497053 4234 raft_consensus.cc:2454] T 91ff8be30dc04f2fa844e8e32e0c46f4 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:02.497568 7090 leader_election.cc:302] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db0742ee3860468590eefef6ae6074fd, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:02.497967 7551 raft_consensus.cc:2781] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:02.498081 7551 raft_consensus.cc:482] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:02.498168 7551 raft_consensus.cc:3037] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:02.501255 7551 raft_consensus.cc:504] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:02.503712 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ff8be30dc04f2fa844e8e32e0c46f4" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:02.503965 7022 raft_consensus.cc:3037] T 91ff8be30dc04f2fa844e8e32e0c46f4 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:02.503949 7516 consensus_queue.cc:1035] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:02.507314 7022 raft_consensus.cc:2454] T 91ff8be30dc04f2fa844e8e32e0c46f4 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:02.507911 7090 leader_election.cc:302] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db0742ee3860468590eefef6ae6074fd, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:02.509332 7551 leader_election.cc:288] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:02.511197 7551 raft_consensus.cc:2781] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:02.513788 7512 ts_tablet_manager.cc:1888] T 0b14eb56de3940de9fbf03d31b860005 P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:02.516166 4477 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:02.516981 7551 raft_consensus.cc:686] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Becoming Leader. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Running, Role: LEADER
I20240501 20:10:02.517017 7722 ts_tablet_manager.cc:1888] T 0b14eb56de3940de9fbf03d31b860005 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:02.517781 7551 consensus_queue.cc:237] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:02.520396 7722 log.cc:1196] T 0b14eb56de3940de9fbf03d31b860005 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/0b14eb56de3940de9fbf03d31b860005
I20240501 20:10:02.521580 7722 ts_tablet_manager.cc:1909] T 0b14eb56de3940de9fbf03d31b860005 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:02.524358 7722 tablet_replica.cc:330] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:02.524461 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet 0b14eb56de3940de9fbf03d31b860005 (table rebalance_test_table_10 [id=ece9130016fb4578a855bb117efb9bca]) successfully deleted
I20240501 20:10:02.525307 7512 log.cc:1196] T 0b14eb56de3940de9fbf03d31b860005 P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/0b14eb56de3940de9fbf03d31b860005
I20240501 20:10:02.526346 7512 ts_tablet_manager.cc:1909] T 0b14eb56de3940de9fbf03d31b860005 P 10b47782558c41ee91f5e164a372049a: Deleting consensus metadata
I20240501 20:10:02.527976 7722 raft_consensus.cc:2229] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:02.529541 7722 raft_consensus.cc:2258] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:02.531100 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ff8be30dc04f2fa844e8e32e0c46f4" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:10:02.531334 4234 raft_consensus.cc:3037] T 91ff8be30dc04f2fa844e8e32e0c46f4 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:02.532791 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet 0b14eb56de3940de9fbf03d31b860005 (table rebalance_test_table_10 [id=ece9130016fb4578a855bb117efb9bca]) successfully deleted
I20240501 20:10:02.533905 7722 ts_tablet_manager.cc:1875] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:02.534317 4234 raft_consensus.cc:2454] T 91ff8be30dc04f2fa844e8e32e0c46f4 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:02.535513 7512 tablet_replica.cc:330] T 11c38f7be4804540806d743e1fa47b48 P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:02.536013 7512 raft_consensus.cc:2229] T 11c38f7be4804540806d743e1fa47b48 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:02.536342 7512 raft_consensus.cc:2258] T 11c38f7be4804540806d743e1fa47b48 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:02.537626 7770 consensus_queue.cc:1035] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:02.539413 7400 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:10:02.548103 7512 ts_tablet_manager.cc:1875] T 11c38f7be4804540806d743e1fa47b48 P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:02.548509 7495 consensus_queue.cc:1035] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:02.553758 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b14eb56de3940de9fbf03d31b860005 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:02 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:02.554118 7519 tablet_replica.cc:330] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:02.554368 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet 8afb9c6ed6f946eeb7a015a0f6f0c658 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:02 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:02.554435 7519 raft_consensus.cc:2229] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:02.554914 7519 raft_consensus.cc:2258] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:02.554963 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet 11c38f7be4804540806d743e1fa47b48 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:02 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:02.555727 7519 ts_tablet_manager.cc:1875] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:02.557520 7002 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:10:02.560137 4092 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:02.560993 7722 ts_tablet_manager.cc:1888] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:02.563175 7722 log.cc:1196] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/8afb9c6ed6f946eeb7a015a0f6f0c658
I20240501 20:10:02.563709 7722 ts_tablet_manager.cc:1909] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:02.564262 7512 ts_tablet_manager.cc:1888] T 11c38f7be4804540806d743e1fa47b48 P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:02.565424 7512 log.cc:1196] T 11c38f7be4804540806d743e1fa47b48 P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/11c38f7be4804540806d743e1fa47b48
I20240501 20:10:02.566390 7512 ts_tablet_manager.cc:1909] T 11c38f7be4804540806d743e1fa47b48 P 10b47782558c41ee91f5e164a372049a: Deleting consensus metadata
I20240501 20:10:02.566395 7519 ts_tablet_manager.cc:1888] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:02.566529 7519 log.cc:1196] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/0b14eb56de3940de9fbf03d31b860005
I20240501 20:10:02.567117 7722 tablet_replica.cc:330] T 11c38f7be4804540806d743e1fa47b48 P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:02.567729 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet 8afb9c6ed6f946eeb7a015a0f6f0c658 (table rebalance_test_table_10 [id=ece9130016fb4578a855bb117efb9bca]) successfully deleted
I20240501 20:10:02.568373 7722 raft_consensus.cc:2229] T 11c38f7be4804540806d743e1fa47b48 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:02.568714 7722 raft_consensus.cc:2258] T 11c38f7be4804540806d743e1fa47b48 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:02.569252 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet 11c38f7be4804540806d743e1fa47b48 (table rebalance_test_table_10 [id=ece9130016fb4578a855bb117efb9bca]) successfully deleted
I20240501 20:10:02.570569 7722 ts_tablet_manager.cc:1875] T 11c38f7be4804540806d743e1fa47b48 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:02.571723 4610 tablet_service.cc:1467] Processing CreateTablet for tablet f4f03f407a364b80b1925418d0bdbc5d (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f56b02450c6f407b827d0131f37c9ffa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:02.572639 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f03f407a364b80b1925418d0bdbc5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:02.572875 7267 tablet_service.cc:1467] Processing CreateTablet for tablet f82a24aabb2b4386b5916653a97e548c (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f56b02450c6f407b827d0131f37c9ffa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:02.573395 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f82a24aabb2b4386b5916653a97e548c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:02.575500 7002 tablet_service.cc:1467] Processing CreateTablet for tablet f4f03f407a364b80b1925418d0bdbc5d (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f56b02450c6f407b827d0131f37c9ffa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:02.575956 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f03f407a364b80b1925418d0bdbc5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:02.576459 4215 tablet_service.cc:1467] Processing CreateTablet for tablet f4f03f407a364b80b1925418d0bdbc5d (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f56b02450c6f407b827d0131f37c9ffa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:02.576887 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f03f407a364b80b1925418d0bdbc5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:02.581085 7133 tablet_service.cc:1467] Processing CreateTablet for tablet 35c843bebb484f6cb655c146a02261ba (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f56b02450c6f407b827d0131f37c9ffa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:02.581547 7133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35c843bebb484f6cb655c146a02261ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:02.583573 7400 tablet_service.cc:1467] Processing CreateTablet for tablet f82a24aabb2b4386b5916653a97e548c (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f56b02450c6f407b827d0131f37c9ffa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:02.583986 7744 tablet_bootstrap.cc:492] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:02.584021 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f82a24aabb2b4386b5916653a97e548c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:02.585583 7744 tablet_bootstrap.cc:654] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:02.586544 7495 consensus_queue.cc:1035] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:02.587960 7748 tablet_bootstrap.cc:492] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:02.590112 7748 tablet_bootstrap.cc:654] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:02.593634 7519 ts_tablet_manager.cc:1909] T 0b14eb56de3940de9fbf03d31b860005 P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:02.595016 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet 0b14eb56de3940de9fbf03d31b860005 (table rebalance_test_table_10 [id=ece9130016fb4578a855bb117efb9bca]) successfully deleted
I20240501 20:10:02.598347 7519 tablet_replica.cc:330] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:02.598816 7519 raft_consensus.cc:2229] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:02.599628 7519 raft_consensus.cc:2258] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:02.601374 7519 ts_tablet_manager.cc:1875] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:02.608278 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 35c843bebb484f6cb655c146a02261ba (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f56b02450c6f407b827d0131f37c9ffa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:02.608799 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35c843bebb484f6cb655c146a02261ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:02.609122 7860 tablet_bootstrap.cc:492] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:02.610848 7861 tablet_bootstrap.cc:492] T f4f03f407a364b80b1925418d0bdbc5d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:02.612740 7808 tablet_bootstrap.cc:492] T f4f03f407a364b80b1925418d0bdbc5d P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:02.614439 7808 tablet_bootstrap.cc:654] T f4f03f407a364b80b1925418d0bdbc5d P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:02.614495 7748 tablet_bootstrap.cc:492] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:02.614492 7744 tablet_bootstrap.cc:492] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:02.614658 7744 ts_tablet_manager.cc:1367] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.008s
I20240501 20:10:02.614666 7748 ts_tablet_manager.cc:1367] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.005s
I20240501 20:10:02.615536 7744 raft_consensus.cc:348] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:02.615550 7748 raft_consensus.cc:348] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:02.615810 7748 raft_consensus.cc:374] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:02.615860 7744 raft_consensus.cc:374] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:02.615978 7748 raft_consensus.cc:729] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:02.616024 7744 raft_consensus.cc:729] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:02.616350 7744 consensus_queue.cc:260] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:02.616360 7748 consensus_queue.cc:260] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
W20240501 20:10:02.616451 4302 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.68:45205, user_credentials={real_user=slave}} blocked reactor thread for 57268.6us
I20240501 20:10:02.617094 7748 ts_tablet_manager.cc:1398] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:10:02.617098 7744 ts_tablet_manager.cc:1398] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:02.620237 7001 tablet_service.cc:1467] Processing CreateTablet for tablet f82a24aabb2b4386b5916653a97e548c (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f56b02450c6f407b827d0131f37c9ffa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:02.620777 7001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f82a24aabb2b4386b5916653a97e548c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:02.621284 7000 tablet_service.cc:1467] Processing CreateTablet for tablet 35c843bebb484f6cb655c146a02261ba (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=f56b02450c6f407b827d0131f37c9ffa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:02.622046 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35c843bebb484f6cb655c146a02261ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:02.627552 7860 tablet_bootstrap.cc:654] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:02.654547 7861 tablet_bootstrap.cc:654] T f4f03f407a364b80b1925418d0bdbc5d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
W20240501 20:10:02.662336 4301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.65:42941, user_credentials={real_user=slave}} blocked reactor thread for 54338.4us
I20240501 20:10:02.613545 7722 ts_tablet_manager.cc:1888] T 11c38f7be4804540806d743e1fa47b48 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:02.663056 7722 log.cc:1196] T 11c38f7be4804540806d743e1fa47b48 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/11c38f7be4804540806d743e1fa47b48
I20240501 20:10:02.663888 7722 ts_tablet_manager.cc:1909] T 11c38f7be4804540806d743e1fa47b48 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:02.665843 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet 11c38f7be4804540806d743e1fa47b48 (table rebalance_test_table_10 [id=ece9130016fb4578a855bb117efb9bca]) successfully deleted
I20240501 20:10:02.668852 4215 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:02.675591 7519 ts_tablet_manager.cc:1888] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:02.675772 7519 log.cc:1196] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/8afb9c6ed6f946eeb7a015a0f6f0c658
I20240501 20:10:02.676301 7519 ts_tablet_manager.cc:1909] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:02.677407 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet 8afb9c6ed6f946eeb7a015a0f6f0c658 (table rebalance_test_table_10 [id=ece9130016fb4578a855bb117efb9bca]) successfully deleted
I20240501 20:10:02.679224 7860 tablet_bootstrap.cc:492] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:02.679843 7860 ts_tablet_manager.cc:1367] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.071s user 0.004s sys 0.003s
I20240501 20:10:02.680716 7860 raft_consensus.cc:348] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:02.681038 7860 raft_consensus.cc:374] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:02.681190 7860 raft_consensus.cc:729] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:02.681525 7860 consensus_queue.cc:260] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:02.683087 7861 tablet_bootstrap.cc:492] T f4f03f407a364b80b1925418d0bdbc5d P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:02.683287 7861 ts_tablet_manager.cc:1367] T f4f03f407a364b80b1925418d0bdbc5d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.073s user 0.002s sys 0.004s
I20240501 20:10:02.685760 7861 raft_consensus.cc:348] T f4f03f407a364b80b1925418d0bdbc5d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:02.686057 7861 raft_consensus.cc:374] T f4f03f407a364b80b1925418d0bdbc5d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:02.686237 7861 raft_consensus.cc:729] T f4f03f407a364b80b1925418d0bdbc5d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:02.686517 7861 consensus_queue.cc:260] T f4f03f407a364b80b1925418d0bdbc5d P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:02.690315 7861 ts_tablet_manager.cc:1398] T f4f03f407a364b80b1925418d0bdbc5d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:10:02.700672 4092 catalog_manager.cc:5484] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd reported cstate change: term changed from 0 to 1, leader changed from <none> to db0742ee3860468590eefef6ae6074fd (127.29.219.70). New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:02.696156 7860 ts_tablet_manager.cc:1398] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240501 20:10:02.708523 7519 tablet_replica.cc:330] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:02.708843 7519 raft_consensus.cc:2229] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:02.709275 7519 raft_consensus.cc:2258] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:02.710093 7519 ts_tablet_manager.cc:1875] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:02.717747 7863 tablet_bootstrap.cc:492] T f82a24aabb2b4386b5916653a97e548c P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:02.719653 7863 tablet_bootstrap.cc:654] T f82a24aabb2b4386b5916653a97e548c P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:02.724076 7519 ts_tablet_manager.cc:1888] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:02.724220 7519 log.cc:1196] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/11c38f7be4804540806d743e1fa47b48
I20240501 20:10:02.724647 7519 ts_tablet_manager.cc:1909] T 11c38f7be4804540806d743e1fa47b48 P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:02.724748 7847 tablet_replica.cc:330] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:02.725123 7847 raft_consensus.cc:2229] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:02.725452 7847 raft_consensus.cc:2258] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:02.725792 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet 11c38f7be4804540806d743e1fa47b48 (table rebalance_test_table_10 [id=ece9130016fb4578a855bb117efb9bca]) successfully deleted
I20240501 20:10:02.727581 7847 ts_tablet_manager.cc:1875] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:02.750766 7205 raft_consensus.cc:482] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:02.751011 7205 raft_consensus.cc:504] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:02.751895 7205 leader_election.cc:288] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:02.753083 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f03f407a364b80b1925418d0bdbc5d" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:02.753437 4627 raft_consensus.cc:2454] T f4f03f407a364b80b1925418d0bdbc5d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:02.754076 4170 leader_election.cc:302] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:02.754455 7205 raft_consensus.cc:2781] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:02.754590 7205 raft_consensus.cc:482] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:02.754760 7205 raft_consensus.cc:3037] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:02.763643 7205 raft_consensus.cc:504] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:02.765209 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f03f407a364b80b1925418d0bdbc5d" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:10:02.765406 4627 raft_consensus.cc:3037] T f4f03f407a364b80b1925418d0bdbc5d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:02.769438 4627 raft_consensus.cc:2454] T f4f03f407a364b80b1925418d0bdbc5d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:02.770056 4170 leader_election.cc:302] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:02.770385 7657 raft_consensus.cc:2781] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:02.786571 7205 leader_election.cc:288] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:02.787950 7863 tablet_bootstrap.cc:492] T f82a24aabb2b4386b5916653a97e548c P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:02.788152 7863 ts_tablet_manager.cc:1367] T f82a24aabb2b4386b5916653a97e548c P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.071s user 0.000s sys 0.003s
I20240501 20:10:02.790223 7863 raft_consensus.cc:348] T f82a24aabb2b4386b5916653a97e548c P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:02.793800 7863 raft_consensus.cc:374] T f82a24aabb2b4386b5916653a97e548c P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:02.794595 7863 raft_consensus.cc:729] T f82a24aabb2b4386b5916653a97e548c P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:02.795573 7863 consensus_queue.cc:260] T f82a24aabb2b4386b5916653a97e548c P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:02.797420 7863 ts_tablet_manager.cc:1398] T f82a24aabb2b4386b5916653a97e548c P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.009s user 0.000s sys 0.003s
I20240501 20:10:02.798161 7863 tablet_bootstrap.cc:492] T 35c843bebb484f6cb655c146a02261ba P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:02.803980 7847 ts_tablet_manager.cc:1888] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:02.804275 7847 log.cc:1196] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/8afb9c6ed6f946eeb7a015a0f6f0c658
I20240501 20:10:02.805091 7847 ts_tablet_manager.cc:1909] T 8afb9c6ed6f946eeb7a015a0f6f0c658 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting consensus metadata
I20240501 20:10:02.806152 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet 8afb9c6ed6f946eeb7a015a0f6f0c658 (table rebalance_test_table_10 [id=ece9130016fb4578a855bb117efb9bca]) successfully deleted
I20240501 20:10:02.812558 7657 raft_consensus.cc:686] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:10:02.813463 7657 consensus_queue.cc:237] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:02.823042 7863 tablet_bootstrap.cc:654] T 35c843bebb484f6cb655c146a02261ba P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:02.839975 7618 raft_consensus.cc:482] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:02.840441 4091 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:02.840173 7618 raft_consensus.cc:504] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:02.842967 7618 leader_election.cc:288] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:02.847820 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f82a24aabb2b4386b5916653a97e548c" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:02.852387 7420 raft_consensus.cc:2454] T f82a24aabb2b4386b5916653a97e548c P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:02.852748 4235 raft_consensus.cc:1261] T 91ff8be30dc04f2fa844e8e32e0c46f4 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:02.852996 7223 leader_election.cc:302] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:02.853446 7551 consensus_queue.cc:1035] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:02.860185 7618 raft_consensus.cc:2781] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:02.860339 7618 raft_consensus.cc:482] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:02.860430 7618 raft_consensus.cc:3037] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:02.871531 7618 raft_consensus.cc:504] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:02.873003 7618 leader_election.cc:288] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:02.874343 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f82a24aabb2b4386b5916653a97e548c" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:02.874650 7420 raft_consensus.cc:3037] T f82a24aabb2b4386b5916653a97e548c P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:02.877606 7420 raft_consensus.cc:2454] T f82a24aabb2b4386b5916653a97e548c P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:02.878051 7223 leader_election.cc:302] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:02.878518 7618 raft_consensus.cc:2781] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:02.878434 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f03f407a364b80b1925418d0bdbc5d" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:02.879091 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f03f407a364b80b1925418d0bdbc5d" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
W20240501 20:10:02.879165 4169 leader_election.cc:341] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:10:02.879724 4169 leader_election.cc:341] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:02.883831 7022 raft_consensus.cc:1261] T 91ff8be30dc04f2fa844e8e32e0c46f4 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:02.886669 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f82a24aabb2b4386b5916653a97e548c" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:02.890264 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f82a24aabb2b4386b5916653a97e548c" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:02.891727 7863 tablet_bootstrap.cc:492] T 35c843bebb484f6cb655c146a02261ba P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:02.893543 7551 consensus_queue.cc:1035] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:10:02.894503 7222 leader_election.cc:341] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:10:02.894953 7222 leader_election.cc:341] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:02.895507 7808 tablet_bootstrap.cc:492] T f4f03f407a364b80b1925418d0bdbc5d P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:02.899278 7618 raft_consensus.cc:686] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Becoming Leader. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Running, Role: LEADER
I20240501 20:10:02.900585 7618 consensus_queue.cc:237] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:02.903399 7808 ts_tablet_manager.cc:1367] T f4f03f407a364b80b1925418d0bdbc5d P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.291s user 0.007s sys 0.015s
I20240501 20:10:02.903839 7863 ts_tablet_manager.cc:1367] T 35c843bebb484f6cb655c146a02261ba P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.106s user 0.005s sys 0.000s
I20240501 20:10:02.903980 4091 catalog_manager.cc:5576] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237) does not have the latest schema for tablet 91ff8be30dc04f2fa844e8e32e0c46f4 (table rebalancer_dynamic_table [id=8dc02548b7cf43e782b8568293e87f1e]). Expected version 1 got 0
I20240501 20:10:02.905256 7808 raft_consensus.cc:348] T f4f03f407a364b80b1925418d0bdbc5d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:02.906572 7863 raft_consensus.cc:348] T 35c843bebb484f6cb655c146a02261ba P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:02.909550 7863 raft_consensus.cc:374] T 35c843bebb484f6cb655c146a02261ba P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:02.911625 7890 tablet.cc:1722] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:02.912413 7890 tablet.cc:1620] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:10:02.912753 4091 catalog_manager.cc:5484] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:02.916801 7808 raft_consensus.cc:374] T f4f03f407a364b80b1925418d0bdbc5d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:02.917449 7808 raft_consensus.cc:729] T f4f03f407a364b80b1925418d0bdbc5d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:02.917948 7808 consensus_queue.cc:260] T f4f03f407a364b80b1925418d0bdbc5d P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:02.921586 4087 catalog_manager.cc:5484] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71). New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:02.922432 7863 raft_consensus.cc:729] T 35c843bebb484f6cb655c146a02261ba P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:02.923096 7863 consensus_queue.cc:260] T 35c843bebb484f6cb655c146a02261ba P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:02.925230 4092 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=8dc02548b7cf43e782b8568293e87f1e] alter complete (version 1)
I20240501 20:10:02.925479 7808 ts_tablet_manager.cc:1398] T f4f03f407a364b80b1925418d0bdbc5d P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.021s user 0.001s sys 0.002s
I20240501 20:10:02.926947 7808 tablet_bootstrap.cc:492] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:02.930410 7848 raft_consensus.cc:482] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:02.930665 7848 raft_consensus.cc:504] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:02.931875 7848 leader_election.cc:288] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:02.932554 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c843bebb484f6cb655c146a02261ba" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:02.932797 7420 raft_consensus.cc:2454] T 35c843bebb484f6cb655c146a02261ba P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:02.933286 7863 ts_tablet_manager.cc:1398] T 35c843bebb484f6cb655c146a02261ba P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240501 20:10:02.933321 7091 leader_election.cc:302] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:02.933318 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c843bebb484f6cb655c146a02261ba" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:02.933717 7848 raft_consensus.cc:2781] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:02.933820 7848 raft_consensus.cc:482] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:02.933892 7848 raft_consensus.cc:3037] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:02.935531 7808 tablet_bootstrap.cc:654] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
W20240501 20:10:02.936208 7090 leader_election.cc:341] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:02.936736 7848 raft_consensus.cc:504] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:02.937645 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c843bebb484f6cb655c146a02261ba" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
W20240501 20:10:02.938084 7090 leader_election.cc:341] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:02.941821 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c843bebb484f6cb655c146a02261ba" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:02.942046 7420 raft_consensus.cc:3037] T 35c843bebb484f6cb655c146a02261ba P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:02.944938 7420 raft_consensus.cc:2454] T 35c843bebb484f6cb655c146a02261ba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:02.946329 7091 leader_election.cc:302] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters: ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:02.946734 7551 raft_consensus.cc:2781] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:02.946893 7551 raft_consensus.cc:686] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Becoming Leader. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Running, Role: LEADER
I20240501 20:10:02.947162 7551 consensus_queue.cc:237] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:02.962059 4091 catalog_manager.cc:5484] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd reported cstate change: term changed from 0 to 1, leader changed from <none> to db0742ee3860468590eefef6ae6074fd (127.29.219.70). New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:02.963465 7848 leader_election.cc:288] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:02.968739 7891 tablet.cc:1722] T 91ff8be30dc04f2fa844e8e32e0c46f4 P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:02.969513 7891 tablet.cc:1620] T 91ff8be30dc04f2fa844e8e32e0c46f4 P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:02.972748 4091 catalog_manager.cc:5576] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237) does not have the latest schema for tablet 91ff8be30dc04f2fa844e8e32e0c46f4 (table rebalancer_dynamic_table [id=8dc02548b7cf43e782b8568293e87f1e]). Expected version 1 got 0
I20240501 20:10:02.975286 7808 tablet_bootstrap.cc:492] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:02.975492 7808 ts_tablet_manager.cc:1367] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.002s
I20240501 20:10:02.976514 7808 raft_consensus.cc:348] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:02.976902 7808 raft_consensus.cc:374] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:02.977221 7808 raft_consensus.cc:729] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:02.977627 7808 consensus_queue.cc:260] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:02.980749 7808 ts_tablet_manager.cc:1398] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:10:02.984150 4091 catalog_manager.cc:5576] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941) does not have the latest schema for tablet 91ff8be30dc04f2fa844e8e32e0c46f4 (table rebalancer_dynamic_table [id=8dc02548b7cf43e782b8568293e87f1e]). Expected version 1 got 0
I20240501 20:10:02.984668 7808 tablet_bootstrap.cc:492] T 35c843bebb484f6cb655c146a02261ba P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:02.984915 7884 mvcc.cc:205] Tried to move back new op lower bound from 7022977854877945856 to 7022977853589798912. Current Snapshot: MvccSnapshot[applied={T|T < 7022977854877945856}]
I20240501 20:10:02.985337 7896 tablet.cc:1722] T 91ff8be30dc04f2fa844e8e32e0c46f4 P dca57823003e4729b769ca12367d7530: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:02.986608 7896 tablet.cc:1620] T 91ff8be30dc04f2fa844e8e32e0c46f4 P dca57823003e4729b769ca12367d7530: MemRowSet was empty: no flush needed.
I20240501 20:10:02.992715 7808 tablet_bootstrap.cc:654] T 35c843bebb484f6cb655c146a02261ba P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:02.995002 4087 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240501 20:10:02.995281 4087 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240501 20:10:02.998821 4087 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 1d03fbe032b344d98d8f500c2667dd44
I20240501 20:10:02.999392 4087 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet c78926d6cd5d44c6a19ee07396984b61
I20240501 20:10:02.999722 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d03fbe032b344d98d8f500c2667dd44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:02 UTC) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:02.999877 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d03fbe032b344d98d8f500c2667dd44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:02 UTC) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:03.000144 7847 tablet_replica.cc:330] T 1d03fbe032b344d98d8f500c2667dd44 P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:03.000200 7512 tablet_replica.cc:330] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:03.000375 7847 raft_consensus.cc:2229] T 1d03fbe032b344d98d8f500c2667dd44 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:03.000398 7512 raft_consensus.cc:2229] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:03.000453 4087 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 6bf9f8a78da54553bfe6eaa1f3b28a3f
I20240501 20:10:03.000619 7847 raft_consensus.cc:2258] T 1d03fbe032b344d98d8f500c2667dd44 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:03.000953 7512 raft_consensus.cc:2258] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:03.001384 7847 ts_tablet_manager.cc:1875] T 1d03fbe032b344d98d8f500c2667dd44 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.001710 7512 ts_tablet_manager.cc:1875] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.001793 7808 tablet_bootstrap.cc:492] T 35c843bebb484f6cb655c146a02261ba P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:03.002053 7808 ts_tablet_manager.cc:1367] T 35c843bebb484f6cb655c146a02261ba P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.001s
I20240501 20:10:03.002769 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d03fbe032b344d98d8f500c2667dd44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:02 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:03.002822 7808 raft_consensus.cc:348] T 35c843bebb484f6cb655c146a02261ba P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:03.003034 7808 raft_consensus.cc:374] T 35c843bebb484f6cb655c146a02261ba P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.003618 7808 raft_consensus.cc:729] T 35c843bebb484f6cb655c146a02261ba P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.003876 7808 consensus_queue.cc:260] T 35c843bebb484f6cb655c146a02261ba P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:03.004431 7808 ts_tablet_manager.cc:1398] T 35c843bebb484f6cb655c146a02261ba P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:03.006596 7519 tablet_replica.cc:330] T 1d03fbe032b344d98d8f500c2667dd44 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:03.006575 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bf9f8a78da54553bfe6eaa1f3b28a3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:02 UTC) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:03.006718 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet c78926d6cd5d44c6a19ee07396984b61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:02 UTC) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:03.006898 7519 raft_consensus.cc:2229] T 1d03fbe032b344d98d8f500c2667dd44 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:03.007066 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet c78926d6cd5d44c6a19ee07396984b61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:02 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:03.007066 4213 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bf9f8a78da54553bfe6eaa1f3b28a3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:02 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:03.007184 7519 raft_consensus.cc:2258] T 1d03fbe032b344d98d8f500c2667dd44 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:03.007866 7519 ts_tablet_manager.cc:1875] T 1d03fbe032b344d98d8f500c2667dd44 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.008018 4609 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bf9f8a78da54553bfe6eaa1f3b28a3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:02 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:03.008358 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet c78926d6cd5d44c6a19ee07396984b61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:02 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:03.008603 7722 tablet_replica.cc:330] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:03.008745 7722 raft_consensus.cc:2229] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:03.008949 7722 raft_consensus.cc:2258] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:03.009512 7722 ts_tablet_manager.cc:1875] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.013119 7512 ts_tablet_manager.cc:1888] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:03.013235 7512 log.cc:1196] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/1d03fbe032b344d98d8f500c2667dd44
I20240501 20:10:03.013653 7512 ts_tablet_manager.cc:1909] T 1d03fbe032b344d98d8f500c2667dd44 P 10b47782558c41ee91f5e164a372049a: Deleting consensus metadata
I20240501 20:10:03.014633 7512 tablet_replica.cc:330] T c78926d6cd5d44c6a19ee07396984b61 P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:03.014843 7512 raft_consensus.cc:2229] T c78926d6cd5d44c6a19ee07396984b61 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:03.014864 7519 ts_tablet_manager.cc:1888] T 1d03fbe032b344d98d8f500c2667dd44 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:03.014999 7519 log.cc:1196] T 1d03fbe032b344d98d8f500c2667dd44 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/1d03fbe032b344d98d8f500c2667dd44
I20240501 20:10:03.015051 7512 raft_consensus.cc:2258] T c78926d6cd5d44c6a19ee07396984b61 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:03.015372 7519 ts_tablet_manager.cc:1909] T 1d03fbe032b344d98d8f500c2667dd44 P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:03.015769 7512 ts_tablet_manager.cc:1875] T c78926d6cd5d44c6a19ee07396984b61 P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.016278 7519 tablet_replica.cc:330] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:03.016454 7519 raft_consensus.cc:2229] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:03.016777 7519 raft_consensus.cc:2258] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:03.017410 7519 ts_tablet_manager.cc:1875] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.020009 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet 1d03fbe032b344d98d8f500c2667dd44 (table rebalance_test_table_12 [id=c3c3c4ff4c8444a38dae5dc10a57d879]) successfully deleted
I20240501 20:10:03.020649 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet 1d03fbe032b344d98d8f500c2667dd44 (table rebalance_test_table_12 [id=c3c3c4ff4c8444a38dae5dc10a57d879]) successfully deleted
I20240501 20:10:03.023674 7512 ts_tablet_manager.cc:1888] T c78926d6cd5d44c6a19ee07396984b61 P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:03.024032 7512 log.cc:1196] T c78926d6cd5d44c6a19ee07396984b61 P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/c78926d6cd5d44c6a19ee07396984b61
I20240501 20:10:03.024078 7519 ts_tablet_manager.cc:1888] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:03.024223 7519 log.cc:1196] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/c78926d6cd5d44c6a19ee07396984b61
I20240501 20:10:03.024519 7512 ts_tablet_manager.cc:1909] T c78926d6cd5d44c6a19ee07396984b61 P 10b47782558c41ee91f5e164a372049a: Deleting consensus metadata
I20240501 20:10:03.024742 7519 ts_tablet_manager.cc:1909] T c78926d6cd5d44c6a19ee07396984b61 P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:03.025913 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet c78926d6cd5d44c6a19ee07396984b61 (table rebalance_test_table_12 [id=c3c3c4ff4c8444a38dae5dc10a57d879]) successfully deleted
I20240501 20:10:03.026321 7519 tablet_replica.cc:330] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:03.026454 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet c78926d6cd5d44c6a19ee07396984b61 (table rebalance_test_table_12 [id=c3c3c4ff4c8444a38dae5dc10a57d879]) successfully deleted
I20240501 20:10:03.026669 7519 raft_consensus.cc:2229] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:03.027004 7519 raft_consensus.cc:2258] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:03.027750 7519 ts_tablet_manager.cc:1875] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.034005 7519 ts_tablet_manager.cc:1888] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:03.034155 7519 log.cc:1196] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/6bf9f8a78da54553bfe6eaa1f3b28a3f
I20240501 20:10:03.034559 7519 ts_tablet_manager.cc:1909] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:03.035537 7847 ts_tablet_manager.cc:1888] T 1d03fbe032b344d98d8f500c2667dd44 P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:03.035645 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet 6bf9f8a78da54553bfe6eaa1f3b28a3f (table rebalance_test_table_12 [id=c3c3c4ff4c8444a38dae5dc10a57d879]) successfully deleted
I20240501 20:10:03.035717 7722 ts_tablet_manager.cc:1888] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:03.035732 7847 log.cc:1196] T 1d03fbe032b344d98d8f500c2667dd44 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/1d03fbe032b344d98d8f500c2667dd44
I20240501 20:10:03.035866 7722 log.cc:1196] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/6bf9f8a78da54553bfe6eaa1f3b28a3f
I20240501 20:10:03.036211 7722 ts_tablet_manager.cc:1909] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:03.036229 7847 ts_tablet_manager.cc:1909] T 1d03fbe032b344d98d8f500c2667dd44 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting consensus metadata
I20240501 20:10:03.037032 7722 tablet_replica.cc:330] T c78926d6cd5d44c6a19ee07396984b61 P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:03.037030 7847 tablet_replica.cc:330] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:03.037184 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet 1d03fbe032b344d98d8f500c2667dd44 (table rebalance_test_table_12 [id=c3c3c4ff4c8444a38dae5dc10a57d879]) successfully deleted
I20240501 20:10:03.037207 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet 6bf9f8a78da54553bfe6eaa1f3b28a3f (table rebalance_test_table_12 [id=c3c3c4ff4c8444a38dae5dc10a57d879]) successfully deleted
I20240501 20:10:03.037267 7847 raft_consensus.cc:2229] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:03.037283 7722 raft_consensus.cc:2229] T c78926d6cd5d44c6a19ee07396984b61 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:03.037509 7722 raft_consensus.cc:2258] T c78926d6cd5d44c6a19ee07396984b61 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:03.037576 7847 raft_consensus.cc:2258] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:03.038156 7847 ts_tablet_manager.cc:1875] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.038159 7722 ts_tablet_manager.cc:1875] T c78926d6cd5d44c6a19ee07396984b61 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.042719 7847 ts_tablet_manager.cc:1888] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:03.042896 7847 log.cc:1196] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/6bf9f8a78da54553bfe6eaa1f3b28a3f
I20240501 20:10:03.043304 7847 ts_tablet_manager.cc:1909] T 6bf9f8a78da54553bfe6eaa1f3b28a3f P 3e974c03d3c546f9ab6c91e762a5426b: Deleting consensus metadata
I20240501 20:10:03.044209 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet 6bf9f8a78da54553bfe6eaa1f3b28a3f (table rebalance_test_table_12 [id=c3c3c4ff4c8444a38dae5dc10a57d879]) successfully deleted
I20240501 20:10:03.044284 7722 ts_tablet_manager.cc:1888] T c78926d6cd5d44c6a19ee07396984b61 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:03.044462 7722 log.cc:1196] T c78926d6cd5d44c6a19ee07396984b61 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/c78926d6cd5d44c6a19ee07396984b61
I20240501 20:10:03.044893 7722 ts_tablet_manager.cc:1909] T c78926d6cd5d44c6a19ee07396984b61 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:03.045866 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet c78926d6cd5d44c6a19ee07396984b61 (table rebalance_test_table_12 [id=c3c3c4ff4c8444a38dae5dc10a57d879]) successfully deleted
I20240501 20:10:03.054661 4087 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:03.061857 7890 tablet.cc:1722] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:03.062338 7890 tablet.cc:1620] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:10:03.064344 7896 tablet.cc:1722] T 91ff8be30dc04f2fa844e8e32e0c46f4 P dca57823003e4729b769ca12367d7530: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:03.064800 7896 tablet.cc:1620] T 91ff8be30dc04f2fa844e8e32e0c46f4 P dca57823003e4729b769ca12367d7530: MemRowSet was empty: no flush needed.
I20240501 20:10:03.067328 7891 tablet.cc:1722] T 91ff8be30dc04f2fa844e8e32e0c46f4 P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:03.067791 7891 tablet.cc:1620] T 91ff8be30dc04f2fa844e8e32e0c46f4 P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:03.071992 4087 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=8dc02548b7cf43e782b8568293e87f1e] alter complete (version 2)
I20240501 20:10:03.120097 7880 raft_consensus.cc:482] T f4f03f407a364b80b1925418d0bdbc5d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:03.120337 7880 raft_consensus.cc:504] T f4f03f407a364b80b1925418d0bdbc5d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:03.121201 7880 leader_election.cc:288] T f4f03f407a364b80b1925418d0bdbc5d P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:03.121789 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f03f407a364b80b1925418d0bdbc5d" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:03.129647 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f03f407a364b80b1925418d0bdbc5d" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:03.130003 4627 raft_consensus.cc:2379] T f4f03f407a364b80b1925418d0bdbc5d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef0a4ada548446b3a531cb39a3c4e38d in current term 1: Already voted for candidate dca57823003e4729b769ca12367d7530 in this term.
I20240501 20:10:03.129927 4366 consensus_queue.cc:237] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:03.130913 6958 leader_election.cc:302] T f4f03f407a364b80b1925418d0bdbc5d P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef0a4ada548446b3a531cb39a3c4e38d; no voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530
I20240501 20:10:03.131268 7880 raft_consensus.cc:3037] T f4f03f407a364b80b1925418d0bdbc5d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.133656 4497 raft_consensus.cc:1261] T c31148e9e4dd42c3a11787d9384c79c2 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:03.134104 7880 raft_consensus.cc:2726] T f4f03f407a364b80b1925418d0bdbc5d P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:03.134673 7856 consensus_queue.cc:1035] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:03.139693 4627 raft_consensus.cc:1261] T c31148e9e4dd42c3a11787d9384c79c2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:03.139201 7862 raft_consensus.cc:2932] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:03.140794 4497 raft_consensus.cc:2932] T c31148e9e4dd42c3a11787d9384c79c2 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:03.146293 7862 consensus_queue.cc:1035] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:03.147591 4087 catalog_manager.cc:5484] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:10:03.160838 4302 consensus_peers.cc:485] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a -> Peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): Couldn't send request to peer b9957d4086bc4a76b18ed9265a5739d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c31148e9e4dd42c3a11787d9384c79c2. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:03.163091 4627 raft_consensus.cc:2932] T c31148e9e4dd42c3a11787d9384c79c2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:03.175670 4091 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:03.184285 7890 tablet.cc:1722] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:03.184962 7896 tablet.cc:1722] T 91ff8be30dc04f2fa844e8e32e0c46f4 P dca57823003e4729b769ca12367d7530: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:03.184952 7891 tablet.cc:1722] T 91ff8be30dc04f2fa844e8e32e0c46f4 P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:03.196966 4497 consensus_queue.cc:237] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:03.195380 4091 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table_new_name [id=8dc02548b7cf43e782b8568293e87f1e] alter complete (version 3)
I20240501 20:10:03.204353 4627 raft_consensus.cc:1261] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:03.205052 7767 consensus_queue.cc:1035] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:03.205890 4366 raft_consensus.cc:1261] T 46df0faa681b469abe9e383b181a6053 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:03.208637 7767 consensus_queue.cc:1035] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:10:03.215345 4434 consensus_peers.cc:485] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46df0faa681b469abe9e383b181a6053. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:03.220240 7767 raft_consensus.cc:2932] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:03.221132 4366 raft_consensus.cc:2932] T 46df0faa681b469abe9e383b181a6053 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:03.228083 4087 catalog_manager.cc:5484] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:03.233580 4627 raft_consensus.cc:2932] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:03.253623 4497 consensus_queue.cc:237] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:03.256594 4235 raft_consensus.cc:1261] T 66c533992f8b4c7a937e79f397636bcc P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:03.257351 4365 raft_consensus.cc:1261] T 66c533992f8b4c7a937e79f397636bcc P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:03.257351 7919 consensus_queue.cc:1035] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:03.257946 7767 consensus_queue.cc:1035] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:03.261127 7919 raft_consensus.cc:2932] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:03.263875 4365 raft_consensus.cc:2932] T 66c533992f8b4c7a937e79f397636bcc P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:03.265515 4235 raft_consensus.cc:2932] T 66c533992f8b4c7a937e79f397636bcc P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:03.270938 4087 catalog_manager.cc:5484] T 66c533992f8b4c7a937e79f397636bcc P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
W20240501 20:10:03.278319 4432 consensus_peers.cc:485] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b -> Peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Couldn't send request to peer db0742ee3860468590eefef6ae6074fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 66c533992f8b4c7a937e79f397636bcc. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:03.299883 4235 consensus_queue.cc:237] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:03.300940 4091 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:03.301163 4091 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:03.303423 4627 raft_consensus.cc:1261] T dfaa3794205d482c90aa4f860fa99cf3 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:03.303443 4168 consensus_peers.cc:485] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 -> Peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Couldn't send request to peer bceeae1c79964acabdae8ff771bf0c97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dfaa3794205d482c90aa4f860fa99cf3. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:03.303969 7657 consensus_queue.cc:1035] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:03.304128 4497 raft_consensus.cc:1261] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:03.304730 7657 consensus_queue.cc:1035] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:03.305012 4091 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 91ff8be30dc04f2fa844e8e32e0c46f4
I20240501 20:10:03.305915 4213 tablet_service.cc:1514] Processing DeleteTablet for tablet 91ff8be30dc04f2fa844e8e32e0c46f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:03 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:03.306197 7000 tablet_service.cc:1514] Processing DeleteTablet for tablet 91ff8be30dc04f2fa844e8e32e0c46f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:03 UTC) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:03.306423 7519 tablet_replica.cc:330] T 91ff8be30dc04f2fa844e8e32e0c46f4 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:03.306639 7519 raft_consensus.cc:2229] T 91ff8be30dc04f2fa844e8e32e0c46f4 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:03.306589 7133 tablet_service.cc:1514] Processing DeleteTablet for tablet 91ff8be30dc04f2fa844e8e32e0c46f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:03 UTC) from {username='slave'} at 127.0.0.1:52036
I20240501 20:10:03.306816 7941 tablet_replica.cc:330] T 91ff8be30dc04f2fa844e8e32e0c46f4 P ef0a4ada548446b3a531cb39a3c4e38d: stopping tablet replica
I20240501 20:10:03.306872 7519 raft_consensus.cc:2258] T 91ff8be30dc04f2fa844e8e32e0c46f4 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:03.307027 7941 raft_consensus.cc:2229] T 91ff8be30dc04f2fa844e8e32e0c46f4 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:03.307303 7941 raft_consensus.cc:2258] T 91ff8be30dc04f2fa844e8e32e0c46f4 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:03.308358 7519 ts_tablet_manager.cc:1875] T 91ff8be30dc04f2fa844e8e32e0c46f4 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.310245 7941 ts_tablet_manager.cc:1875] T 91ff8be30dc04f2fa844e8e32e0c46f4 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.311177 7880 raft_consensus.cc:482] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:03.311364 7880 raft_consensus.cc:504] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:03.312104 7880 leader_election.cc:288] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:03.312887 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f82a24aabb2b4386b5916653a97e548c" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:03.313185 7420 raft_consensus.cc:2379] T f82a24aabb2b4386b5916653a97e548c P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef0a4ada548446b3a531cb39a3c4e38d in current term 1: Already voted for candidate bceeae1c79964acabdae8ff771bf0c97 in this term.
I20240501 20:10:03.314064 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f82a24aabb2b4386b5916653a97e548c" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:03.315769 6958 leader_election.cc:302] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef0a4ada548446b3a531cb39a3c4e38d; no voters: b9957d4086bc4a76b18ed9265a5739d5, bceeae1c79964acabdae8ff771bf0c97
I20240501 20:10:03.317414 7880 raft_consensus.cc:3037] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.317718 7519 ts_tablet_manager.cc:1888] T 91ff8be30dc04f2fa844e8e32e0c46f4 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:03.317844 7519 log.cc:1196] T 91ff8be30dc04f2fa844e8e32e0c46f4 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/91ff8be30dc04f2fa844e8e32e0c46f4
I20240501 20:10:03.318285 7519 ts_tablet_manager.cc:1909] T 91ff8be30dc04f2fa844e8e32e0c46f4 P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:03.319336 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet 91ff8be30dc04f2fa844e8e32e0c46f4 (table rebalancer_dynamic_table_new_name [id=8dc02548b7cf43e782b8568293e87f1e]) successfully deleted
I20240501 20:10:03.320396 7941 ts_tablet_manager.cc:1888] T 91ff8be30dc04f2fa844e8e32e0c46f4 P ef0a4ada548446b3a531cb39a3c4e38d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:03.320580 7880 raft_consensus.cc:2726] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:03.321661 7941 log.cc:1196] T 91ff8be30dc04f2fa844e8e32e0c46f4 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/wals/91ff8be30dc04f2fa844e8e32e0c46f4
I20240501 20:10:03.322877 7941 ts_tablet_manager.cc:1909] T 91ff8be30dc04f2fa844e8e32e0c46f4 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting consensus metadata
I20240501 20:10:03.324343 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet 91ff8be30dc04f2fa844e8e32e0c46f4 (table rebalancer_dynamic_table_new_name [id=8dc02548b7cf43e782b8568293e87f1e]) successfully deleted
I20240501 20:10:03.330385 7943 tablet_replica.cc:330] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd: stopping tablet replica
I20240501 20:10:03.330610 7943 raft_consensus.cc:2229] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:03.331209 7943 raft_consensus.cc:2258] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:03.332224 7943 ts_tablet_manager.cc:1875] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.334184 7879 raft_consensus.cc:2932] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:03.334986 4627 raft_consensus.cc:2932] T dfaa3794205d482c90aa4f860fa99cf3 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:03.338850 7943 ts_tablet_manager.cc:1888] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:03.339035 7943 log.cc:1196] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/wals/91ff8be30dc04f2fa844e8e32e0c46f4
I20240501 20:10:03.339040 4498 raft_consensus.cc:2932] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:03.339684 7943 ts_tablet_manager.cc:1909] T 91ff8be30dc04f2fa844e8e32e0c46f4 P db0742ee3860468590eefef6ae6074fd: Deleting consensus metadata
I20240501 20:10:03.340698 4076 catalog_manager.cc:4830] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): tablet 91ff8be30dc04f2fa844e8e32e0c46f4 (table rebalancer_dynamic_table_new_name [id=8dc02548b7cf43e782b8568293e87f1e]) successfully deleted
I20240501 20:10:03.343676 4091 catalog_manager.cc:5484] T dfaa3794205d482c90aa4f860fa99cf3 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:03.344455 7205 consensus_queue.cc:1035] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:03.349521 7657 consensus_queue.cc:1035] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:03.381745 4235 consensus_queue.cc:237] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:03.384681 4498 raft_consensus.cc:1261] T 34cb2f3c90cc4158a61f426c0e5cb86b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:03.385186 7657 consensus_queue.cc:1035] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:03.385424 4365 raft_consensus.cc:1261] T 34cb2f3c90cc4158a61f426c0e5cb86b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:03.385502 4170 consensus_peers.cc:485] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 -> Peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): Couldn't send request to peer b9957d4086bc4a76b18ed9265a5739d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34cb2f3c90cc4158a61f426c0e5cb86b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:03.385962 7657 consensus_queue.cc:1035] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:03.387845 7880 raft_consensus.cc:482] T 35c843bebb484f6cb655c146a02261ba P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:03.388018 7880 raft_consensus.cc:504] T 35c843bebb484f6cb655c146a02261ba P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:03.388749 7880 leader_election.cc:288] T 35c843bebb484f6cb655c146a02261ba P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:03.388069 7205 raft_consensus.cc:2932] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:03.389181 7154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c843bebb484f6cb655c146a02261ba" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:03.391093 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c843bebb484f6cb655c146a02261ba" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:03.391387 7420 raft_consensus.cc:2379] T 35c843bebb484f6cb655c146a02261ba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef0a4ada548446b3a531cb39a3c4e38d in current term 1: Already voted for candidate db0742ee3860468590eefef6ae6074fd in this term.
I20240501 20:10:03.392004 6958 leader_election.cc:302] T 35c843bebb484f6cb655c146a02261ba P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef0a4ada548446b3a531cb39a3c4e38d; no voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd
I20240501 20:10:03.391928 4498 raft_consensus.cc:2932] T 34cb2f3c90cc4158a61f426c0e5cb86b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:03.392669 7880 raft_consensus.cc:3037] T 35c843bebb484f6cb655c146a02261ba P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.395695 7880 raft_consensus.cc:2726] T 35c843bebb484f6cb655c146a02261ba P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:03.395510 4365 raft_consensus.cc:2932] T 34cb2f3c90cc4158a61f426c0e5cb86b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:03.397897 7848 consensus_queue.cc:1035] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:03.402289 4091 catalog_manager.cc:5484] T 34cb2f3c90cc4158a61f426c0e5cb86b P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:03.407955 4087 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:03.414065 7848 consensus_queue.cc:1035] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:03.421803 7133 tablet_service.cc:1467] Processing CreateTablet for tablet 0b81e56bfb4c4e93ae68516d9a70538d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c1beed3afaf146648e40929ee5cc2a29]), partition=
I20240501 20:10:03.422235 7133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b81e56bfb4c4e93ae68516d9a70538d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.422256 7000 tablet_service.cc:1467] Processing CreateTablet for tablet 0b81e56bfb4c4e93ae68516d9a70538d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c1beed3afaf146648e40929ee5cc2a29]), partition=
I20240501 20:10:03.422668 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b81e56bfb4c4e93ae68516d9a70538d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.427065 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 0b81e56bfb4c4e93ae68516d9a70538d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c1beed3afaf146648e40929ee5cc2a29]), partition=
I20240501 20:10:03.427440 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b81e56bfb4c4e93ae68516d9a70538d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.428905 7618 consensus_queue.cc:1035] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:03.435814 7808 tablet_bootstrap.cc:492] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:03.437954 7808 tablet_bootstrap.cc:654] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.446828 7966 tablet_bootstrap.cc:492] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:03.444136 7965 tablet_bootstrap.cc:492] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:03.451481 4235 consensus_queue.cc:237] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:03.456923 7966 tablet_bootstrap.cc:654] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.457129 7965 tablet_bootstrap.cc:654] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.457340 7808 tablet_bootstrap.cc:492] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:03.457626 7808 ts_tablet_manager.cc:1367] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.004s
I20240501 20:10:03.458500 7808 raft_consensus.cc:348] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:03.459482 7808 raft_consensus.cc:374] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.459583 7808 raft_consensus.cc:729] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.460726 7808 consensus_queue.cc:260] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:03.461469 7808 ts_tablet_manager.cc:1398] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 20:10:03.463199 4365 raft_consensus.cc:1261] T 9d634c1feb7449f686cf3af2a07181e0 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:03.463825 7955 consensus_queue.cc:1035] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:10:03.464704 4169 consensus_peers.cc:485] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d634c1feb7449f686cf3af2a07181e0. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:03.471474 4498 raft_consensus.cc:1261] T 9d634c1feb7449f686cf3af2a07181e0 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:03.471441 7618 consensus_queue.cc:1035] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:03.472131 7955 consensus_queue.cc:1035] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:03.478489 7966 tablet_bootstrap.cc:492] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:03.478788 7966 ts_tablet_manager.cc:1367] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.002s
I20240501 20:10:03.479830 7966 raft_consensus.cc:348] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:03.480554 7966 raft_consensus.cc:374] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.480679 7966 raft_consensus.cc:729] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.480966 7966 consensus_queue.cc:260] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:03.483889 7965 tablet_bootstrap.cc:492] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:03.484163 7965 ts_tablet_manager.cc:1367] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.000s
I20240501 20:10:03.485244 7965 raft_consensus.cc:348] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:03.485631 7965 raft_consensus.cc:374] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.485785 7965 raft_consensus.cc:729] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.486075 7965 consensus_queue.cc:260] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:03.487077 7965 ts_tablet_manager.cc:1398] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:03.488899 7966 ts_tablet_manager.cc:1398] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20240501 20:10:03.489718 7955 raft_consensus.cc:2932] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:03.495088 4365 raft_consensus.cc:2932] T 9d634c1feb7449f686cf3af2a07181e0 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:03.496618 4091 catalog_manager.cc:5484] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:03.500162 4498 raft_consensus.cc:2932] T 9d634c1feb7449f686cf3af2a07181e0 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:03.510784 4091 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240501 20:10:03.511090 4091 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240501 20:10:03.515004 4091 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet b0dfb22caa4f4855b4976eb3cb27c681
I20240501 20:10:03.515817 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet b0dfb22caa4f4855b4976eb3cb27c681 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:03 UTC) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:03.516116 7512 tablet_replica.cc:330] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:03.516157 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet b0dfb22caa4f4855b4976eb3cb27c681 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:03 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:03.516191 4213 tablet_service.cc:1514] Processing DeleteTablet for tablet b0dfb22caa4f4855b4976eb3cb27c681 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:03 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:03.516341 7512 raft_consensus.cc:2229] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:03.516443 7519 tablet_replica.cc:330] T b0dfb22caa4f4855b4976eb3cb27c681 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:03.516466 4091 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet a215d2ce50c347efb5b536a27d037094
I20240501 20:10:03.516681 7519 raft_consensus.cc:2229] T b0dfb22caa4f4855b4976eb3cb27c681 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:03.516713 7512 raft_consensus.cc:2258] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:03.516928 7519 raft_consensus.cc:2258] T b0dfb22caa4f4855b4976eb3cb27c681 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:03.517131 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet a215d2ce50c347efb5b536a27d037094 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:03 UTC) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:03.517486 7512 ts_tablet_manager.cc:1875] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.517918 7847 tablet_replica.cc:330] T a215d2ce50c347efb5b536a27d037094 P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:03.517920 7519 ts_tablet_manager.cc:1875] T b0dfb22caa4f4855b4976eb3cb27c681 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.518150 7847 raft_consensus.cc:2229] T a215d2ce50c347efb5b536a27d037094 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:03.518373 7847 raft_consensus.cc:2258] T a215d2ce50c347efb5b536a27d037094 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:03.519112 7847 ts_tablet_manager.cc:1875] T a215d2ce50c347efb5b536a27d037094 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.522783 7519 ts_tablet_manager.cc:1888] T b0dfb22caa4f4855b4976eb3cb27c681 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:03.522804 7722 tablet_replica.cc:330] T b0dfb22caa4f4855b4976eb3cb27c681 P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:03.522897 7519 log.cc:1196] T b0dfb22caa4f4855b4976eb3cb27c681 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/b0dfb22caa4f4855b4976eb3cb27c681
I20240501 20:10:03.523036 7722 raft_consensus.cc:2229] T b0dfb22caa4f4855b4976eb3cb27c681 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:03.523255 7519 ts_tablet_manager.cc:1909] T b0dfb22caa4f4855b4976eb3cb27c681 P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:03.523291 7722 raft_consensus.cc:2258] T b0dfb22caa4f4855b4976eb3cb27c681 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:03.524008 7722 ts_tablet_manager.cc:1875] T b0dfb22caa4f4855b4976eb3cb27c681 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.525043 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet b0dfb22caa4f4855b4976eb3cb27c681 (table rebalance_test_table_14 [id=62c829b3c0b3400bb175722fa826b0f4]) successfully deleted
I20240501 20:10:03.527056 4091 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 8add007651394975a012f04fbf7241ed
I20240501 20:10:03.527516 7512 ts_tablet_manager.cc:1888] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:03.527740 7512 log.cc:1196] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/b0dfb22caa4f4855b4976eb3cb27c681
I20240501 20:10:03.528353 7512 ts_tablet_manager.cc:1909] T b0dfb22caa4f4855b4976eb3cb27c681 P 10b47782558c41ee91f5e164a372049a: Deleting consensus metadata
I20240501 20:10:03.528643 4609 tablet_service.cc:1514] Processing DeleteTablet for tablet 8add007651394975a012f04fbf7241ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:03 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:03.528640 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet a215d2ce50c347efb5b536a27d037094 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:03 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:03.529700 4213 tablet_service.cc:1514] Processing DeleteTablet for tablet a215d2ce50c347efb5b536a27d037094 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:03 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:03.529963 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet 8add007651394975a012f04fbf7241ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:03 UTC) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:03.530279 7519 tablet_replica.cc:330] T a215d2ce50c347efb5b536a27d037094 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:03.530050 4092 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:03.530560 7519 raft_consensus.cc:2229] T a215d2ce50c347efb5b536a27d037094 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:03.530799 7512 tablet_replica.cc:330] T 8add007651394975a012f04fbf7241ed P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:03.530889 7519 raft_consensus.cc:2258] T a215d2ce50c347efb5b536a27d037094 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:03.531129 7512 raft_consensus.cc:2229] T 8add007651394975a012f04fbf7241ed P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:03.531262 7722 ts_tablet_manager.cc:1888] T b0dfb22caa4f4855b4976eb3cb27c681 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:03.531394 7512 raft_consensus.cc:2258] T 8add007651394975a012f04fbf7241ed P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:03.531451 7722 log.cc:1196] T b0dfb22caa4f4855b4976eb3cb27c681 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/b0dfb22caa4f4855b4976eb3cb27c681
I20240501 20:10:03.531704 7519 ts_tablet_manager.cc:1875] T a215d2ce50c347efb5b536a27d037094 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.532114 7722 ts_tablet_manager.cc:1909] T b0dfb22caa4f4855b4976eb3cb27c681 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:03.532238 7512 ts_tablet_manager.cc:1875] T 8add007651394975a012f04fbf7241ed P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.529732 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet 8add007651394975a012f04fbf7241ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:03 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:03.533152 7722 tablet_replica.cc:330] T 8add007651394975a012f04fbf7241ed P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:03.533413 7722 raft_consensus.cc:2229] T 8add007651394975a012f04fbf7241ed P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:03.533636 7722 raft_consensus.cc:2258] T 8add007651394975a012f04fbf7241ed P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:03.534369 7722 ts_tablet_manager.cc:1875] T 8add007651394975a012f04fbf7241ed P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.535707 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet b0dfb22caa4f4855b4976eb3cb27c681 (table rebalance_test_table_14 [id=62c829b3c0b3400bb175722fa826b0f4]) successfully deleted
I20240501 20:10:03.535964 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet b0dfb22caa4f4855b4976eb3cb27c681 (table rebalance_test_table_14 [id=62c829b3c0b3400bb175722fa826b0f4]) successfully deleted
I20240501 20:10:03.538774 7519 ts_tablet_manager.cc:1888] T a215d2ce50c347efb5b536a27d037094 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:03.538930 7519 log.cc:1196] T a215d2ce50c347efb5b536a27d037094 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/a215d2ce50c347efb5b536a27d037094
I20240501 20:10:03.539415 7519 ts_tablet_manager.cc:1909] T a215d2ce50c347efb5b536a27d037094 P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:03.539770 7847 ts_tablet_manager.cc:1888] T a215d2ce50c347efb5b536a27d037094 P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:03.539975 7847 log.cc:1196] T a215d2ce50c347efb5b536a27d037094 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/a215d2ce50c347efb5b536a27d037094
I20240501 20:10:03.540556 7519 tablet_replica.cc:330] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:03.540728 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet a215d2ce50c347efb5b536a27d037094 (table rebalance_test_table_14 [id=62c829b3c0b3400bb175722fa826b0f4]) successfully deleted
I20240501 20:10:03.541391 7519 raft_consensus.cc:2229] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:03.541582 7847 ts_tablet_manager.cc:1909] T a215d2ce50c347efb5b536a27d037094 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting consensus metadata
I20240501 20:10:03.541826 7519 raft_consensus.cc:2258] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:03.543233 7133 tablet_service.cc:1467] Processing CreateTablet for tablet 422e49e6312c48a2ab779701c35ed10e (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c0709b4853a54a8ea0cd4c5dfb289ba1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:03.543442 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet a215d2ce50c347efb5b536a27d037094 (table rebalance_test_table_14 [id=62c829b3c0b3400bb175722fa826b0f4]) successfully deleted
I20240501 20:10:03.543411 7400 tablet_service.cc:1467] Processing CreateTablet for tablet 422e49e6312c48a2ab779701c35ed10e (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c0709b4853a54a8ea0cd4c5dfb289ba1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:03.543689 7133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 422e49e6312c48a2ab779701c35ed10e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.543861 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 422e49e6312c48a2ab779701c35ed10e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.547086 7000 tablet_service.cc:1467] Processing CreateTablet for tablet 422e49e6312c48a2ab779701c35ed10e (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c0709b4853a54a8ea0cd4c5dfb289ba1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:03.547549 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 422e49e6312c48a2ab779701c35ed10e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.548669 7512 ts_tablet_manager.cc:1888] T 8add007651394975a012f04fbf7241ed P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:03.548825 7512 log.cc:1196] T 8add007651394975a012f04fbf7241ed P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/8add007651394975a012f04fbf7241ed
I20240501 20:10:03.549297 7512 ts_tablet_manager.cc:1909] T 8add007651394975a012f04fbf7241ed P 10b47782558c41ee91f5e164a372049a: Deleting consensus metadata
I20240501 20:10:03.549608 7134 tablet_service.cc:1467] Processing CreateTablet for tablet 1283fda7ece44e199d74712fd8bba9da (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c0709b4853a54a8ea0cd4c5dfb289ba1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:03.550002 7134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1283fda7ece44e199d74712fd8bba9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.550490 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet 8add007651394975a012f04fbf7241ed (table rebalance_test_table_14 [id=62c829b3c0b3400bb175722fa826b0f4]) successfully deleted
I20240501 20:10:03.551823 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 1283fda7ece44e199d74712fd8bba9da (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c0709b4853a54a8ea0cd4c5dfb289ba1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:03.552209 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1283fda7ece44e199d74712fd8bba9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.552124 7267 tablet_service.cc:1467] Processing CreateTablet for tablet 1283fda7ece44e199d74712fd8bba9da (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c0709b4853a54a8ea0cd4c5dfb289ba1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:03.552589 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1283fda7ece44e199d74712fd8bba9da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.559207 7808 tablet_bootstrap.cc:492] T 422e49e6312c48a2ab779701c35ed10e P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:03.560328 4215 tablet_service.cc:1467] Processing CreateTablet for tablet f44675b6874a423398de361c7ccfd583 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c0709b4853a54a8ea0cd4c5dfb289ba1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:03.560771 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f44675b6874a423398de361c7ccfd583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.560918 7808 tablet_bootstrap.cc:654] T 422e49e6312c48a2ab779701c35ed10e P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.562074 7966 tablet_bootstrap.cc:492] T 1283fda7ece44e199d74712fd8bba9da P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:03.563719 7966 tablet_bootstrap.cc:654] T 1283fda7ece44e199d74712fd8bba9da P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.564461 7519 ts_tablet_manager.cc:1875] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.570509 7965 tablet_bootstrap.cc:492] T 1283fda7ece44e199d74712fd8bba9da P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:03.571938 7266 tablet_service.cc:1467] Processing CreateTablet for tablet f44675b6874a423398de361c7ccfd583 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c0709b4853a54a8ea0cd4c5dfb289ba1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:03.572402 7266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f44675b6874a423398de361c7ccfd583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.572422 7965 tablet_bootstrap.cc:654] T 1283fda7ece44e199d74712fd8bba9da P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.575271 4347 tablet_service.cc:1467] Processing CreateTablet for tablet f44675b6874a423398de361c7ccfd583 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=c0709b4853a54a8ea0cd4c5dfb289ba1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:03.575795 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f44675b6874a423398de361c7ccfd583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.577942 7966 tablet_bootstrap.cc:492] T 1283fda7ece44e199d74712fd8bba9da P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:03.578189 7966 ts_tablet_manager.cc:1367] T 1283fda7ece44e199d74712fd8bba9da P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.004s
I20240501 20:10:03.579154 7966 raft_consensus.cc:348] T 1283fda7ece44e199d74712fd8bba9da P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:03.579490 7980 tablet_bootstrap.cc:492] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:03.579525 7966 raft_consensus.cc:374] T 1283fda7ece44e199d74712fd8bba9da P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.579681 7966 raft_consensus.cc:729] T 1283fda7ece44e199d74712fd8bba9da P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.580072 7966 consensus_queue.cc:260] T 1283fda7ece44e199d74712fd8bba9da P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:03.586521 7808 tablet_bootstrap.cc:492] T 422e49e6312c48a2ab779701c35ed10e P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:03.586683 7722 ts_tablet_manager.cc:1888] T 8add007651394975a012f04fbf7241ed P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:03.586736 7808 ts_tablet_manager.cc:1367] T 422e49e6312c48a2ab779701c35ed10e P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.009s
I20240501 20:10:03.587002 7722 log.cc:1196] T 8add007651394975a012f04fbf7241ed P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/8add007651394975a012f04fbf7241ed
I20240501 20:10:03.587649 7722 ts_tablet_manager.cc:1909] T 8add007651394975a012f04fbf7241ed P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:03.587495 7808 raft_consensus.cc:348] T 422e49e6312c48a2ab779701c35ed10e P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:03.588809 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet 8add007651394975a012f04fbf7241ed (table rebalance_test_table_14 [id=62c829b3c0b3400bb175722fa826b0f4]) successfully deleted
I20240501 20:10:03.588972 7966 ts_tablet_manager.cc:1398] T 1283fda7ece44e199d74712fd8bba9da P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 20:10:03.590368 7722 tablet_replica.cc:330] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:03.590658 7722 raft_consensus.cc:2229] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:03.591091 7722 raft_consensus.cc:2258] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:03.591879 7722 ts_tablet_manager.cc:1875] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:03.593075 7966 tablet_bootstrap.cc:492] T 422e49e6312c48a2ab779701c35ed10e P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:03.596884 7966 tablet_bootstrap.cc:654] T 422e49e6312c48a2ab779701c35ed10e P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.596930 7808 raft_consensus.cc:374] T 422e49e6312c48a2ab779701c35ed10e P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.597064 7808 raft_consensus.cc:729] T 422e49e6312c48a2ab779701c35ed10e P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.597457 7808 consensus_queue.cc:260] T 422e49e6312c48a2ab779701c35ed10e P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:03.600281 7808 ts_tablet_manager.cc:1398] T 422e49e6312c48a2ab779701c35ed10e P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240501 20:10:03.600541 7986 ts_tablet_manager.cc:902] T c31148e9e4dd42c3a11787d9384c79c2 P b9957d4086bc4a76b18ed9265a5739d5: Initiating tablet copy from peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:03.601866 7519 ts_tablet_manager.cc:1888] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:03.600956 7985 tablet_bootstrap.cc:492] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:03.602036 7519 log.cc:1196] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/8add007651394975a012f04fbf7241ed
I20240501 20:10:03.605634 7519 ts_tablet_manager.cc:1909] T 8add007651394975a012f04fbf7241ed P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:03.606227 7985 tablet_bootstrap.cc:654] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.607398 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet 8add007651394975a012f04fbf7241ed (table rebalance_test_table_14 [id=62c829b3c0b3400bb175722fa826b0f4]) successfully deleted
I20240501 20:10:03.610836 7987 tablet_bootstrap.cc:492] T f44675b6874a423398de361c7ccfd583 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:03.617188 7987 tablet_bootstrap.cc:654] T f44675b6874a423398de361c7ccfd583 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.618373 7966 tablet_bootstrap.cc:492] T 422e49e6312c48a2ab779701c35ed10e P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:03.619618 7966 ts_tablet_manager.cc:1367] T 422e49e6312c48a2ab779701c35ed10e P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20240501 20:10:03.621258 7966 raft_consensus.cc:348] T 422e49e6312c48a2ab779701c35ed10e P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:03.622855 7722 ts_tablet_manager.cc:1888] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:03.623037 7980 tablet_bootstrap.cc:654] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.623076 7722 log.cc:1196] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/a215d2ce50c347efb5b536a27d037094
I20240501 20:10:03.623502 7966 raft_consensus.cc:374] T 422e49e6312c48a2ab779701c35ed10e P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.623894 7966 raft_consensus.cc:729] T 422e49e6312c48a2ab779701c35ed10e P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.624073 7722 ts_tablet_manager.cc:1909] T a215d2ce50c347efb5b536a27d037094 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:03.625309 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet a215d2ce50c347efb5b536a27d037094 (table rebalance_test_table_14 [id=62c829b3c0b3400bb175722fa826b0f4]) successfully deleted
I20240501 20:10:03.626045 7966 consensus_queue.cc:260] T 422e49e6312c48a2ab779701c35ed10e P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:03.628026 7984 tablet_bootstrap.cc:492] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:03.628927 7965 tablet_bootstrap.cc:492] T 1283fda7ece44e199d74712fd8bba9da P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:03.629079 7985 tablet_bootstrap.cc:492] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:03.629204 7965 ts_tablet_manager.cc:1367] T 1283fda7ece44e199d74712fd8bba9da P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.059s user 0.011s sys 0.011s
I20240501 20:10:03.629269 7985 ts_tablet_manager.cc:1367] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.006s
I20240501 20:10:03.630321 7985 raft_consensus.cc:348] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:03.630569 7985 raft_consensus.cc:374] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.630789 7985 raft_consensus.cc:729] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.631048 7985 consensus_queue.cc:260] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:03.631733 7985 ts_tablet_manager.cc:1398] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:03.632031 7966 ts_tablet_manager.cc:1398] T 422e49e6312c48a2ab779701c35ed10e P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240501 20:10:03.632748 7984 tablet_bootstrap.cc:654] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.644204 7987 tablet_bootstrap.cc:492] T f44675b6874a423398de361c7ccfd583 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:10:03.644476 7987 ts_tablet_manager.cc:1367] T f44675b6874a423398de361c7ccfd583 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.003s
I20240501 20:10:03.645364 7987 raft_consensus.cc:348] T f44675b6874a423398de361c7ccfd583 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:03.645653 7987 raft_consensus.cc:374] T f44675b6874a423398de361c7ccfd583 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.645790 7987 raft_consensus.cc:729] T f44675b6874a423398de361c7ccfd583 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.646060 7848 raft_consensus.cc:482] T 1283fda7ece44e199d74712fd8bba9da P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:03.646220 7848 raft_consensus.cc:504] T 1283fda7ece44e199d74712fd8bba9da P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:03.647060 7848 leader_election.cc:288] T 1283fda7ece44e199d74712fd8bba9da P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:03.648424 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1283fda7ece44e199d74712fd8bba9da" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
W20240501 20:10:03.650099 7089 leader_election.cc:341] T 1283fda7ece44e199d74712fd8bba9da P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:03.630275 7965 raft_consensus.cc:348] T 1283fda7ece44e199d74712fd8bba9da P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:03.654924 7965 raft_consensus.cc:374] T 1283fda7ece44e199d74712fd8bba9da P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.655093 7965 raft_consensus.cc:729] T 1283fda7ece44e199d74712fd8bba9da P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.657928 7965 consensus_queue.cc:260] T 1283fda7ece44e199d74712fd8bba9da P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:03.658540 7998 ts_tablet_manager.cc:902] T 66c533992f8b4c7a937e79f397636bcc P db0742ee3860468590eefef6ae6074fd: Initiating tablet copy from peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:03.661552 7876 raft_consensus.cc:482] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:03.661756 7876 raft_consensus.cc:504] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:03.662372 7965 ts_tablet_manager.cc:1398] T 1283fda7ece44e199d74712fd8bba9da P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.033s user 0.003s sys 0.000s
I20240501 20:10:03.663137 7876 leader_election.cc:288] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:03.664300 7980 tablet_bootstrap.cc:492] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:03.664484 7980 ts_tablet_manager.cc:1367] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.085s user 0.004s sys 0.007s
I20240501 20:10:03.665572 7980 raft_consensus.cc:348] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:03.665864 7980 raft_consensus.cc:374] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.666005 7980 raft_consensus.cc:729] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.666352 7980 consensus_queue.cc:260] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:03.668359 7974 raft_consensus.cc:482] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:03.668495 7974 raft_consensus.cc:504] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:03.669358 7974 leader_election.cc:288] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:03.671418 7154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1283fda7ece44e199d74712fd8bba9da" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:03.671828 7154 raft_consensus.cc:2454] T 1283fda7ece44e199d74712fd8bba9da P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:03.672583 7220 leader_election.cc:302] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:03.672890 7984 tablet_bootstrap.cc:492] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:03.673046 7984 ts_tablet_manager.cc:1367] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.000s
I20240501 20:10:03.674010 7984 raft_consensus.cc:348] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:03.674326 7984 raft_consensus.cc:374] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.674434 7984 raft_consensus.cc:729] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.674743 7984 consensus_queue.cc:260] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:03.675155 7974 raft_consensus.cc:2781] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:03.675314 7974 raft_consensus.cc:482] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:03.675420 7974 raft_consensus.cc:3037] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.675493 7984 ts_tablet_manager.cc:1398] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:03.676514 7980 ts_tablet_manager.cc:1398] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240501 20:10:03.677605 7980 tablet_bootstrap.cc:492] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:03.679375 7980 tablet_bootstrap.cc:654] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:03.680713 7974 raft_consensus.cc:504] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:03.687981 7974 leader_election.cc:288] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Requested vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:03.688426 7154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1283fda7ece44e199d74712fd8bba9da" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:03.689981 7987 consensus_queue.cc:260] T f44675b6874a423398de361c7ccfd583 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:03.690586 7154 raft_consensus.cc:3037] T 1283fda7ece44e199d74712fd8bba9da P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.690977 7987 ts_tablet_manager.cc:1398] T f44675b6874a423398de361c7ccfd583 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.046s user 0.002s sys 0.000s
I20240501 20:10:03.693118 7880 raft_consensus.cc:482] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:03.693300 7880 raft_consensus.cc:504] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:03.693921 7880 leader_election.cc:288] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:03.694995 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b81e56bfb4c4e93ae68516d9a70538d" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:03.695276 7155 raft_consensus.cc:2454] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:03.696080 6955 leader_election.cc:302] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db0742ee3860468590eefef6ae6074fd, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:03.696557 7880 raft_consensus.cc:2781] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:03.696643 7880 raft_consensus.cc:482] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:03.696733 7880 raft_consensus.cc:3037] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.701253 7980 tablet_bootstrap.cc:492] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:03.702925 7880 raft_consensus.cc:504] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:03.704239 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b81e56bfb4c4e93ae68516d9a70538d" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:03.704475 7155 raft_consensus.cc:3037] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.707307 7155 raft_consensus.cc:2454] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:03.707763 6955 leader_election.cc:302] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db0742ee3860468590eefef6ae6074fd, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:03.708168 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b81e56bfb4c4e93ae68516d9a70538d" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:03.708261 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b81e56bfb4c4e93ae68516d9a70538d" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:10:03.708429 4627 raft_consensus.cc:3037] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.710219 7880 leader_election.cc:288] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:03.711159 7880 raft_consensus.cc:2781] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:03.711272 4627 raft_consensus.cc:2454] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:03.712726 7980 ts_tablet_manager.cc:1367] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20240501 20:10:03.712888 7880 raft_consensus.cc:686] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Becoming Leader. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Running, Role: LEADER
I20240501 20:10:03.713327 7880 consensus_queue.cc:237] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:03.714304 7980 raft_consensus.cc:348] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:03.715392 7154 raft_consensus.cc:2454] T 1283fda7ece44e199d74712fd8bba9da P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:03.716665 7980 raft_consensus.cc:374] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:03.723098 7986 tablet_copy_client.cc:308] T c31148e9e4dd42c3a11787d9384c79c2 P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.66:40689
I20240501 20:10:03.727226 7220 leader_election.cc:302] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:03.727650 7974 raft_consensus.cc:2781] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:03.727829 7974 raft_consensus.cc:686] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Becoming Leader. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Running, Role: LEADER
I20240501 20:10:03.728129 7974 consensus_queue.cc:237] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:03.727660 7980 raft_consensus.cc:729] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:03.732939 7980 consensus_queue.cc:260] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:03.734148 7980 ts_tablet_manager.cc:1398] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20240501 20:10:03.734942 7969 raft_consensus.cc:482] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:03.735128 7969 raft_consensus.cc:504] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:03.735926 7969 leader_election.cc:288] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:03.758594 7872 raft_consensus.cc:482] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:03.758793 7872 raft_consensus.cc:504] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:03.759661 7872 leader_election.cc:288] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:03.761023 4388 tablet_copy_service.cc:140] P 10b47782558c41ee91f5e164a372049a: Received BeginTabletCopySession request for tablet c31148e9e4dd42c3a11787d9384c79c2 from peer b9957d4086bc4a76b18ed9265a5739d5 ({username='slave'} at 127.29.219.72:44421)
I20240501 20:10:03.761294 4388 tablet_copy_service.cc:161] P 10b47782558c41ee91f5e164a372049a: Beginning new tablet copy session on tablet c31148e9e4dd42c3a11787d9384c79c2 from peer b9957d4086bc4a76b18ed9265a5739d5 at {username='slave'} at 127.29.219.72:44421: session id = b9957d4086bc4a76b18ed9265a5739d5-c31148e9e4dd42c3a11787d9384c79c2
I20240501 20:10:03.766950 4388 tablet_copy_source_session.cc:205] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:03.766999 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b81e56bfb4c4e93ae68516d9a70538d" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:03.768868 7154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422e49e6312c48a2ab779701c35ed10e" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:03.768908 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b81e56bfb4c4e93ae68516d9a70538d" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:03.769830 7986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c31148e9e4dd42c3a11787d9384c79c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.769985 7154 raft_consensus.cc:2454] T 422e49e6312c48a2ab779701c35ed10e P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:10:03.770231 7155 raft_consensus.cc:2379] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in current term 1: Already voted for candidate ef0a4ada548446b3a531cb39a3c4e38d in this term.
I20240501 20:10:03.771198 7657 raft_consensus.cc:482] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:03.771349 7657 raft_consensus.cc:504] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:03.771407 7353 leader_election.cc:302] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:03.772049 7657 leader_election.cc:288] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:03.772543 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f44675b6874a423398de361c7ccfd583" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:10:03.772783 4365 raft_consensus.cc:2454] T f44675b6874a423398de361c7ccfd583 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:03.772827 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f44675b6874a423398de361c7ccfd583" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:03.773028 7284 raft_consensus.cc:2454] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:03.773229 4169 leader_election.cc:302] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:03.773663 7657 raft_consensus.cc:2781] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:03.773798 7657 raft_consensus.cc:482] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:03.773890 7657 raft_consensus.cc:3037] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.774063 7872 raft_consensus.cc:2781] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:03.774179 7872 raft_consensus.cc:482] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:03.774262 7872 raft_consensus.cc:3037] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.776749 7657 raft_consensus.cc:504] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:03.777944 7872 raft_consensus.cc:504] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:03.778021 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f44675b6874a423398de361c7ccfd583" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:03.778218 7284 raft_consensus.cc:3037] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.779448 7872 leader_election.cc:288] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:03.779754 7154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422e49e6312c48a2ab779701c35ed10e" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:03.780031 7154 raft_consensus.cc:3037] T 422e49e6312c48a2ab779701c35ed10e P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.780956 7284 raft_consensus.cc:2454] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:03.782843 4168 leader_election.cc:302] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:03.783174 7879 raft_consensus.cc:2781] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:03.783326 7879 raft_consensus.cc:686] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:10:03.783612 7879 consensus_queue.cc:237] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:03.785722 7657 leader_election.cc:288] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:03.786350 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f44675b6874a423398de361c7ccfd583" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:10:03.786697 4365 raft_consensus.cc:3037] T f44675b6874a423398de361c7ccfd583 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.790000 4562 leader_election.cc:302] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e; no voters: db0742ee3860468590eefef6ae6074fd, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:03.790441 7876 raft_consensus.cc:2726] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:03.791736 7986 tablet_copy_client.cc:783] T c31148e9e4dd42c3a11787d9384c79c2 P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:03.792038 4365 raft_consensus.cc:2454] T f44675b6874a423398de361c7ccfd583 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:03.792124 7986 tablet_copy_client.cc:647] T c31148e9e4dd42c3a11787d9384c79c2 P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:03.795538 7986 tablet_copy_client.cc:523] T c31148e9e4dd42c3a11787d9384c79c2 P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:03.807199 7986 tablet_bootstrap.cc:492] T c31148e9e4dd42c3a11787d9384c79c2 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
W20240501 20:10:03.808786 6955 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.219.70:36321, user_credentials={real_user=slave}} blocked reactor thread for 101041us
I20240501 20:10:03.808914 4091 catalog_manager.cc:5484] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71). New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:03.812520 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422e49e6312c48a2ab779701c35ed10e" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:03.812844 7020 raft_consensus.cc:2454] T 422e49e6312c48a2ab779701c35ed10e P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:10:03.814522 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422e49e6312c48a2ab779701c35ed10e" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:03.814769 7020 raft_consensus.cc:3037] T 422e49e6312c48a2ab779701c35ed10e P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.816483 8024 ts_tablet_manager.cc:902] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d: Initiating tablet copy from peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:03.818406 7020 raft_consensus.cc:2454] T 422e49e6312c48a2ab779701c35ed10e P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:03.821550 7355 leader_election.cc:302] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:03.821974 7872 raft_consensus.cc:2781] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:03.822425 7872 raft_consensus.cc:686] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Becoming Leader. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Running, Role: LEADER
I20240501 20:10:03.822813 7872 consensus_queue.cc:237] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:03.825337 7998 tablet_copy_client.cc:308] T 66c533992f8b4c7a937e79f397636bcc P db0742ee3860468590eefef6ae6074fd: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.67:34595
I20240501 20:10:03.834420 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f44675b6874a423398de361c7ccfd583" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:10:03.835067 4365 raft_consensus.cc:2379] T f44675b6874a423398de361c7ccfd583 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bceeae1c79964acabdae8ff771bf0c97 in current term 1: Already voted for candidate dca57823003e4729b769ca12367d7530 in this term.
I20240501 20:10:03.849673 4091 catalog_manager.cc:5484] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:03.852303 8024 tablet_copy_client.cc:308] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.67:34595
I20240501 20:10:03.866665 8041 ts_tablet_manager.cc:902] T 9d634c1feb7449f686cf3af2a07181e0 P ef0a4ada548446b3a531cb39a3c4e38d: Initiating tablet copy from peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:03.872983 4091 catalog_manager.cc:5484] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72). New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:03.874171 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1283fda7ece44e199d74712fd8bba9da" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:03.874439 4631 raft_consensus.cc:2454] T 1283fda7ece44e199d74712fd8bba9da P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:03.875152 7091 leader_election.cc:302] T 1283fda7ece44e199d74712fd8bba9da P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, db0742ee3860468590eefef6ae6074fd; no voters: bceeae1c79964acabdae8ff771bf0c97
I20240501 20:10:03.875638 7850 raft_consensus.cc:2741] T 1283fda7ece44e199d74712fd8bba9da P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240501 20:10:03.870466 7154 raft_consensus.cc:2454] T 422e49e6312c48a2ab779701c35ed10e P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:03.885002 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1283fda7ece44e199d74712fd8bba9da" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:10:03.885303 4627 raft_consensus.cc:3037] T 1283fda7ece44e199d74712fd8bba9da P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:03.885259 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1283fda7ece44e199d74712fd8bba9da" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:03.888204 4627 raft_consensus.cc:2454] T 1283fda7ece44e199d74712fd8bba9da P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:03.897356 4091 catalog_manager.cc:5484] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69). New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:03.902807 8041 tablet_copy_client.cc:308] T 9d634c1feb7449f686cf3af2a07181e0 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.65:42941
I20240501 20:10:03.904554 8030 ts_tablet_manager.cc:902] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5: Initiating tablet copy from peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:03.905433 8038 ts_tablet_manager.cc:902] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97: Initiating tablet copy from peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:03.907685 4519 tablet_copy_service.cc:140] P 3e974c03d3c546f9ab6c91e762a5426b: Received BeginTabletCopySession request for tablet 66c533992f8b4c7a937e79f397636bcc from peer db0742ee3860468590eefef6ae6074fd ({username='slave'} at 127.29.219.70:56941)
I20240501 20:10:03.908107 4519 tablet_copy_service.cc:161] P 3e974c03d3c546f9ab6c91e762a5426b: Beginning new tablet copy session on tablet 66c533992f8b4c7a937e79f397636bcc from peer db0742ee3860468590eefef6ae6074fd at {username='slave'} at 127.29.219.70:56941: session id = db0742ee3860468590eefef6ae6074fd-66c533992f8b4c7a937e79f397636bcc
I20240501 20:10:03.912966 4519 tablet_copy_source_session.cc:205] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:03.915961 7998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66c533992f8b4c7a937e79f397636bcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.921685 4255 tablet_copy_service.cc:140] P dca57823003e4729b769ca12367d7530: Received BeginTabletCopySession request for tablet 9d634c1feb7449f686cf3af2a07181e0 from peer ef0a4ada548446b3a531cb39a3c4e38d ({username='slave'} at 127.29.219.69:51289)
I20240501 20:10:03.921932 4255 tablet_copy_service.cc:161] P dca57823003e4729b769ca12367d7530: Beginning new tablet copy session on tablet 9d634c1feb7449f686cf3af2a07181e0 from peer ef0a4ada548446b3a531cb39a3c4e38d at {username='slave'} at 127.29.219.69:51289: session id = ef0a4ada548446b3a531cb39a3c4e38d-9d634c1feb7449f686cf3af2a07181e0
I20240501 20:10:03.925819 7998 tablet_copy_client.cc:783] T 66c533992f8b4c7a937e79f397636bcc P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:03.926313 7998 tablet_copy_client.cc:647] T 66c533992f8b4c7a937e79f397636bcc P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:03.926725 4255 tablet_copy_source_session.cc:205] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:03.926875 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f44675b6874a423398de361c7ccfd583" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:03.927559 7222 leader_election.cc:302] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97; no voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530
I20240501 20:10:03.928038 7974 raft_consensus.cc:2726] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:03.932374 7998 tablet_copy_client.cc:523] T 66c533992f8b4c7a937e79f397636bcc P db0742ee3860468590eefef6ae6074fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:03.936899 7998 tablet_bootstrap.cc:492] T 66c533992f8b4c7a937e79f397636bcc P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:03.948321 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d634c1feb7449f686cf3af2a07181e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.956266 8041 tablet_copy_client.cc:783] T 9d634c1feb7449f686cf3af2a07181e0 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:03.956691 8041 tablet_copy_client.cc:647] T 9d634c1feb7449f686cf3af2a07181e0 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:03.959806 4519 tablet_copy_service.cc:140] P 3e974c03d3c546f9ab6c91e762a5426b: Received BeginTabletCopySession request for tablet 46df0faa681b469abe9e383b181a6053 from peer ef0a4ada548446b3a531cb39a3c4e38d ({username='slave'} at 127.29.219.69:60767)
I20240501 20:10:03.960004 4519 tablet_copy_service.cc:161] P 3e974c03d3c546f9ab6c91e762a5426b: Beginning new tablet copy session on tablet 46df0faa681b469abe9e383b181a6053 from peer ef0a4ada548446b3a531cb39a3c4e38d at {username='slave'} at 127.29.219.69:60767: session id = ef0a4ada548446b3a531cb39a3c4e38d-46df0faa681b469abe9e383b181a6053
I20240501 20:10:03.963105 4519 tablet_copy_source_session.cc:205] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:03.965065 8041 tablet_copy_client.cc:523] T 9d634c1feb7449f686cf3af2a07181e0 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:03.968168 8024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46df0faa681b469abe9e383b181a6053. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.974145 8030 tablet_copy_client.cc:308] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.65:42941
I20240501 20:10:03.975028 8024 tablet_copy_client.cc:783] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:03.975275 4255 tablet_copy_service.cc:140] P dca57823003e4729b769ca12367d7530: Received BeginTabletCopySession request for tablet 34cb2f3c90cc4158a61f426c0e5cb86b from peer b9957d4086bc4a76b18ed9265a5739d5 ({username='slave'} at 127.29.219.72:48035)
I20240501 20:10:03.975517 4255 tablet_copy_service.cc:161] P dca57823003e4729b769ca12367d7530: Beginning new tablet copy session on tablet 34cb2f3c90cc4158a61f426c0e5cb86b from peer b9957d4086bc4a76b18ed9265a5739d5 at {username='slave'} at 127.29.219.72:48035: session id = b9957d4086bc4a76b18ed9265a5739d5-34cb2f3c90cc4158a61f426c0e5cb86b
I20240501 20:10:03.977978 8038 tablet_copy_client.cc:308] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.65:42941
I20240501 20:10:03.978570 4255 tablet_copy_source_session.cc:205] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:03.979060 4254 tablet_copy_service.cc:140] P dca57823003e4729b769ca12367d7530: Received BeginTabletCopySession request for tablet dfaa3794205d482c90aa4f860fa99cf3 from peer bceeae1c79964acabdae8ff771bf0c97 ({username='slave'} at 127.29.219.71:58089)
I20240501 20:10:03.979251 4254 tablet_copy_service.cc:161] P dca57823003e4729b769ca12367d7530: Beginning new tablet copy session on tablet dfaa3794205d482c90aa4f860fa99cf3 from peer bceeae1c79964acabdae8ff771bf0c97 at {username='slave'} at 127.29.219.71:58089: session id = bceeae1c79964acabdae8ff771bf0c97-dfaa3794205d482c90aa4f860fa99cf3
I20240501 20:10:03.979357 8024 tablet_copy_client.cc:647] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:03.982461 4254 tablet_copy_source_session.cc:205] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:03.982722 8041 tablet_bootstrap.cc:492] T 9d634c1feb7449f686cf3af2a07181e0 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:03.983776 7986 tablet_bootstrap.cc:492] T c31148e9e4dd42c3a11787d9384c79c2 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:03.984294 7986 tablet_bootstrap.cc:492] T c31148e9e4dd42c3a11787d9384c79c2 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap complete.
I20240501 20:10:03.984625 7986 ts_tablet_manager.cc:1367] T c31148e9e4dd42c3a11787d9384c79c2 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.178s user 0.057s sys 0.017s
I20240501 20:10:03.985531 7986 raft_consensus.cc:348] T c31148e9e4dd42c3a11787d9384c79c2 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:03.985823 7986 raft_consensus.cc:729] T c31148e9e4dd42c3a11787d9384c79c2 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: LEARNER
I20240501 20:10:03.986034 7986 consensus_queue.cc:260] T c31148e9e4dd42c3a11787d9384c79c2 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:03.986719 7986 ts_tablet_manager.cc:1398] T c31148e9e4dd42c3a11787d9384c79c2 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 20:10:03.988018 4388 tablet_copy_service.cc:342] P 10b47782558c41ee91f5e164a372049a: Request end of tablet copy session b9957d4086bc4a76b18ed9265a5739d5-c31148e9e4dd42c3a11787d9384c79c2 received from {username='slave'} at 127.29.219.72:44421
I20240501 20:10:03.988204 4388 tablet_copy_service.cc:433] P 10b47782558c41ee91f5e164a372049a: ending tablet copy session b9957d4086bc4a76b18ed9265a5739d5-c31148e9e4dd42c3a11787d9384c79c2 on tablet c31148e9e4dd42c3a11787d9384c79c2 with peer b9957d4086bc4a76b18ed9265a5739d5
I20240501 20:10:03.989239 8024 tablet_copy_client.cc:523] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:03.990578 8030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34cb2f3c90cc4158a61f426c0e5cb86b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.993723 8024 tablet_bootstrap.cc:492] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:03.995498 8030 tablet_copy_client.cc:783] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:03.996084 8038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfaa3794205d482c90aa4f860fa99cf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:03.999168 8030 tablet_copy_client.cc:647] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:04.003451 8030 tablet_copy_client.cc:523] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:04.007159 8030 tablet_bootstrap.cc:492] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:04.011327 8038 tablet_copy_client.cc:783] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:04.011958 8038 tablet_copy_client.cc:647] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:04.014933 8038 tablet_copy_client.cc:523] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:04.014847 4091 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:04.026543 8038 tablet_bootstrap.cc:492] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:04.029178 4092 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240501 20:10:04.029472 4092 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240501 20:10:04.036494 4092 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 4 replicas of tablet 66c533992f8b4c7a937e79f397636bcc
I20240501 20:10:04.037415 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet 66c533992f8b4c7a937e79f397636bcc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:04 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:04.038353 7519 tablet_replica.cc:330] T 66c533992f8b4c7a937e79f397636bcc P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:04.038507 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet 66c533992f8b4c7a937e79f397636bcc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:04 UTC) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:04.038575 7519 raft_consensus.cc:2229] T 66c533992f8b4c7a937e79f397636bcc P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:04.038805 7512 tablet_replica.cc:330] T 66c533992f8b4c7a937e79f397636bcc P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:04.038929 7519 raft_consensus.cc:2258] T 66c533992f8b4c7a937e79f397636bcc P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:04.038996 7512 raft_consensus.cc:2229] T 66c533992f8b4c7a937e79f397636bcc P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:04.039115 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet 66c533992f8b4c7a937e79f397636bcc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:04 UTC) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:04.039299 7512 raft_consensus.cc:2258] T 66c533992f8b4c7a937e79f397636bcc P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:04.039410 7847 tablet_replica.cc:330] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:04.039669 7847 raft_consensus.cc:2229] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:04.040046 7847 raft_consensus.cc:2258] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:04.041319 7512 ts_tablet_manager.cc:1875] T 66c533992f8b4c7a937e79f397636bcc P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:04.042874 7519 ts_tablet_manager.cc:1875] T 66c533992f8b4c7a937e79f397636bcc P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:04.046896 7847 ts_tablet_manager.cc:1875] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:04.054215 7133 tablet_service.cc:1514] Processing DeleteTablet for tablet 66c533992f8b4c7a937e79f397636bcc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:04 UTC) from {username='slave'} at 127.0.0.1:52036
I20240501 20:10:04.056473 4092 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 5d9602e8417d444b828c6401632bffed
I20240501 20:10:04.056504 7519 ts_tablet_manager.cc:1888] T 66c533992f8b4c7a937e79f397636bcc P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:04.056632 7519 log.cc:1196] T 66c533992f8b4c7a937e79f397636bcc P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/66c533992f8b4c7a937e79f397636bcc
I20240501 20:10:04.057174 7519 ts_tablet_manager.cc:1909] T 66c533992f8b4c7a937e79f397636bcc P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:04.058414 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet 66c533992f8b4c7a937e79f397636bcc (table rebalance_test_table_16 [id=e07c82123bdc4376bae10c48e6e6543a]) successfully deleted
I20240501 20:10:04.063755 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d9602e8417d444b828c6401632bffed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:04 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:04.064064 7722 tablet_replica.cc:330] T 5d9602e8417d444b828c6401632bffed P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:04.064285 7722 raft_consensus.cc:2229] T 5d9602e8417d444b828c6401632bffed P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:04.064532 7722 raft_consensus.cc:2258] T 5d9602e8417d444b828c6401632bffed P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:04.065291 7722 ts_tablet_manager.cc:1875] T 5d9602e8417d444b828c6401632bffed P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:04.072600 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d9602e8417d444b828c6401632bffed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:04 UTC) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:04.074739 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d9602e8417d444b828c6401632bffed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:04 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:04.075130 7519 tablet_replica.cc:330] T 5d9602e8417d444b828c6401632bffed P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:04.075376 7519 raft_consensus.cc:2229] T 5d9602e8417d444b828c6401632bffed P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:04.075574 4092 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet db31c514cd704786b4be2217daaa11d5
I20240501 20:10:04.075616 7519 raft_consensus.cc:2258] T 5d9602e8417d444b828c6401632bffed P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:04.076416 7519 ts_tablet_manager.cc:1875] T 5d9602e8417d444b828c6401632bffed P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:04.076486 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet db31c514cd704786b4be2217daaa11d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:04 UTC) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:04.084729 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet db31c514cd704786b4be2217daaa11d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:04 UTC) from {username='slave'} at 127.0.0.1:58624
W20240501 20:10:04.087353 4301 consensus_peers.cc:485] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:04.092556 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet db31c514cd704786b4be2217daaa11d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:04 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:04.095660 7722 ts_tablet_manager.cc:1888] T 5d9602e8417d444b828c6401632bffed P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:04.097697 7519 ts_tablet_manager.cc:1888] T 5d9602e8417d444b828c6401632bffed P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:04.097925 7519 log.cc:1196] T 5d9602e8417d444b828c6401632bffed P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/5d9602e8417d444b828c6401632bffed
I20240501 20:10:04.102353 7519 ts_tablet_manager.cc:1909] T 5d9602e8417d444b828c6401632bffed P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:04.107885 7512 ts_tablet_manager.cc:1888] T 66c533992f8b4c7a937e79f397636bcc P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:04.107959 7998 tablet_bootstrap.cc:492] T 66c533992f8b4c7a937e79f397636bcc P db0742ee3860468590eefef6ae6074fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:04.108462 7998 tablet_bootstrap.cc:492] T 66c533992f8b4c7a937e79f397636bcc P db0742ee3860468590eefef6ae6074fd: Bootstrap complete.
I20240501 20:10:04.108831 7998 ts_tablet_manager.cc:1367] T 66c533992f8b4c7a937e79f397636bcc P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.172s user 0.064s sys 0.000s
I20240501 20:10:04.109836 7998 raft_consensus.cc:348] T 66c533992f8b4c7a937e79f397636bcc P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:04.110157 7998 raft_consensus.cc:729] T 66c533992f8b4c7a937e79f397636bcc P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: LEARNER
I20240501 20:10:04.111491 7519 tablet_replica.cc:330] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:04.111852 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet 5d9602e8417d444b828c6401632bffed (table rebalance_test_table_16 [id=e07c82123bdc4376bae10c48e6e6543a]) successfully deleted
I20240501 20:10:04.112484 7722 log.cc:1196] T 5d9602e8417d444b828c6401632bffed P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/5d9602e8417d444b828c6401632bffed
I20240501 20:10:04.112890 7519 raft_consensus.cc:2229] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:04.113301 7519 raft_consensus.cc:2258] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:04.110447 7998 consensus_queue.cc:260] T 66c533992f8b4c7a937e79f397636bcc P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:04.114162 7519 ts_tablet_manager.cc:1875] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:04.115898 7722 ts_tablet_manager.cc:1909] T 5d9602e8417d444b828c6401632bffed P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:04.116262 7847 ts_tablet_manager.cc:1888] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:04.116529 7512 log.cc:1196] T 66c533992f8b4c7a937e79f397636bcc P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/66c533992f8b4c7a937e79f397636bcc
I20240501 20:10:04.117713 7847 log.cc:1196] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/66c533992f8b4c7a937e79f397636bcc
I20240501 20:10:04.119046 7847 ts_tablet_manager.cc:1909] T 66c533992f8b4c7a937e79f397636bcc P 3e974c03d3c546f9ab6c91e762a5426b: Deleting consensus metadata
I20240501 20:10:04.120306 7512 ts_tablet_manager.cc:1909] T 66c533992f8b4c7a937e79f397636bcc P 10b47782558c41ee91f5e164a372049a: Deleting consensus metadata
I20240501 20:10:04.120510 7998 ts_tablet_manager.cc:1398] T 66c533992f8b4c7a937e79f397636bcc P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20240501 20:10:04.121935 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet 66c533992f8b4c7a937e79f397636bcc (table rebalance_test_table_16 [id=e07c82123bdc4376bae10c48e6e6543a]) successfully deleted
I20240501 20:10:04.122931 7618 consensus_queue.cc:1035] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:04.123953 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet 5d9602e8417d444b828c6401632bffed (table rebalance_test_table_16 [id=e07c82123bdc4376bae10c48e6e6543a]) successfully deleted
I20240501 20:10:04.122931 4519 tablet_copy_service.cc:342] P 3e974c03d3c546f9ab6c91e762a5426b: Request end of tablet copy session db0742ee3860468590eefef6ae6074fd-66c533992f8b4c7a937e79f397636bcc received from {username='slave'} at 127.29.219.70:56941
I20240501 20:10:04.125335 4519 tablet_copy_service.cc:433] P 3e974c03d3c546f9ab6c91e762a5426b: ending tablet copy session db0742ee3860468590eefef6ae6074fd-66c533992f8b4c7a937e79f397636bcc on tablet 66c533992f8b4c7a937e79f397636bcc with peer db0742ee3860468590eefef6ae6074fd
I20240501 20:10:04.129519 7722 tablet_replica.cc:330] T db31c514cd704786b4be2217daaa11d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:04.130980 7722 raft_consensus.cc:2229] T db31c514cd704786b4be2217daaa11d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:04.133208 7512 tablet_replica.cc:330] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:04.134021 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet 66c533992f8b4c7a937e79f397636bcc (table rebalance_test_table_16 [id=e07c82123bdc4376bae10c48e6e6543a]) successfully deleted
I20240501 20:10:04.134092 7519 ts_tablet_manager.cc:1888] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:04.134255 7519 log.cc:1196] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/db31c514cd704786b4be2217daaa11d5
I20240501 20:10:04.134828 7519 ts_tablet_manager.cc:1909] T db31c514cd704786b4be2217daaa11d5 P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:04.135424 7512 raft_consensus.cc:2229] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:04.136197 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet db31c514cd704786b4be2217daaa11d5 (table rebalance_test_table_16 [id=e07c82123bdc4376bae10c48e6e6543a]) successfully deleted
I20240501 20:10:04.136799 7847 tablet_replica.cc:330] T db31c514cd704786b4be2217daaa11d5 P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:04.134853 7722 raft_consensus.cc:2258] T db31c514cd704786b4be2217daaa11d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:04.141024 7847 raft_consensus.cc:2229] T db31c514cd704786b4be2217daaa11d5 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:04.141801 7512 raft_consensus.cc:2258] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:04.142521 7722 ts_tablet_manager.cc:1875] T db31c514cd704786b4be2217daaa11d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:04.143118 7847 raft_consensus.cc:2258] T db31c514cd704786b4be2217daaa11d5 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:04.144520 7512 ts_tablet_manager.cc:1875] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:04.146157 8024 tablet_bootstrap.cc:492] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:04.146752 8024 tablet_bootstrap.cc:492] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap complete.
I20240501 20:10:04.147042 8024 ts_tablet_manager.cc:1367] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.153s user 0.051s sys 0.010s
I20240501 20:10:04.148082 8024 raft_consensus.cc:348] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:04.148403 8024 raft_consensus.cc:729] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: LEARNER
I20240501 20:10:04.148634 8024 consensus_queue.cc:260] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:04.149228 8024 ts_tablet_manager.cc:1398] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:04.150215 4519 tablet_copy_service.cc:342] P 3e974c03d3c546f9ab6c91e762a5426b: Request end of tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-46df0faa681b469abe9e383b181a6053 received from {username='slave'} at 127.29.219.69:60767
I20240501 20:10:04.160138 4519 tablet_copy_service.cc:433] P 3e974c03d3c546f9ab6c91e762a5426b: ending tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-46df0faa681b469abe9e383b181a6053 on tablet 46df0faa681b469abe9e383b181a6053 with peer ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:04.163734 8060 tablet_replica.cc:330] T 66c533992f8b4c7a937e79f397636bcc P db0742ee3860468590eefef6ae6074fd: stopping tablet replica
I20240501 20:10:04.163730 4631 raft_consensus.cc:1261] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:04.164135 8060 raft_consensus.cc:2229] T 66c533992f8b4c7a937e79f397636bcc P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Raft consensus shutting down.
I20240501 20:10:04.164434 8060 raft_consensus.cc:2258] T 66c533992f8b4c7a937e79f397636bcc P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Raft consensus is shut down!
I20240501 20:10:04.164376 7880 consensus_queue.cc:1035] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:04.165184 7847 ts_tablet_manager.cc:1875] T db31c514cd704786b4be2217daaa11d5 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:04.166133 8060 ts_tablet_manager.cc:1875] T 66c533992f8b4c7a937e79f397636bcc P db0742ee3860468590eefef6ae6074fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:04.166535 7155 raft_consensus.cc:1261] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:04.168932 7880 consensus_queue.cc:1035] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:04.173187 8060 ts_tablet_manager.cc:1888] T 66c533992f8b4c7a937e79f397636bcc P db0742ee3860468590eefef6ae6074fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:04.180838 4092 catalog_manager.cc:5576] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321) does not have the latest schema for tablet 0b81e56bfb4c4e93ae68516d9a70538d (table rebalancer_dynamic_table [id=c1beed3afaf146648e40929ee5cc2a29]). Expected version 1 got 0
I20240501 20:10:04.192724 7974 consensus_queue.cc:1035] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:04.193830 8060 log.cc:1196] T 66c533992f8b4c7a937e79f397636bcc P db0742ee3860468590eefef6ae6074fd: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/wals/66c533992f8b4c7a937e79f397636bcc
I20240501 20:10:04.194482 8060 ts_tablet_manager.cc:1909] T 66c533992f8b4c7a937e79f397636bcc P db0742ee3860468590eefef6ae6074fd: Deleting consensus metadata
I20240501 20:10:04.197029 4076 catalog_manager.cc:4830] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): tablet 66c533992f8b4c7a937e79f397636bcc (table rebalance_test_table_16 [id=e07c82123bdc4376bae10c48e6e6543a]) successfully deleted
I20240501 20:10:04.212317 7022 raft_consensus.cc:1203] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:04.215639 8075 tablet.cc:1722] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:04.220487 4091 catalog_manager.cc:5576] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321) does not have the latest schema for tablet 0b81e56bfb4c4e93ae68516d9a70538d (table rebalancer_dynamic_table [id=c1beed3afaf146648e40929ee5cc2a29]). Expected version 1 got 0
I20240501 20:10:04.224893 7722 ts_tablet_manager.cc:1888] T db31c514cd704786b4be2217daaa11d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:04.229698 8075 tablet.cc:1620] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:10:04.234440 7847 ts_tablet_manager.cc:1888] T db31c514cd704786b4be2217daaa11d5 P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:04.234730 7847 log.cc:1196] T db31c514cd704786b4be2217daaa11d5 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/db31c514cd704786b4be2217daaa11d5
I20240501 20:10:04.239892 7722 log.cc:1196] T db31c514cd704786b4be2217daaa11d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/db31c514cd704786b4be2217daaa11d5
I20240501 20:10:04.239305 8074 tablet.cc:1722] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:04.240437 7512 ts_tablet_manager.cc:1888] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:04.240535 8074 tablet.cc:1620] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:04.240607 4091 catalog_manager.cc:5576] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237) does not have the latest schema for tablet 0b81e56bfb4c4e93ae68516d9a70538d (table rebalancer_dynamic_table [id=c1beed3afaf146648e40929ee5cc2a29]). Expected version 1 got 0
I20240501 20:10:04.241565 7722 ts_tablet_manager.cc:1909] T db31c514cd704786b4be2217daaa11d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:04.245183 7512 log.cc:1196] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/5d9602e8417d444b828c6401632bffed
I20240501 20:10:04.247268 7512 ts_tablet_manager.cc:1909] T 5d9602e8417d444b828c6401632bffed P 10b47782558c41ee91f5e164a372049a: Deleting consensus metadata
I20240501 20:10:04.247676 7847 ts_tablet_manager.cc:1909] T db31c514cd704786b4be2217daaa11d5 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting consensus metadata
I20240501 20:10:04.248914 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet db31c514cd704786b4be2217daaa11d5 (table rebalance_test_table_16 [id=e07c82123bdc4376bae10c48e6e6543a]) successfully deleted
I20240501 20:10:04.250540 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet 5d9602e8417d444b828c6401632bffed (table rebalance_test_table_16 [id=e07c82123bdc4376bae10c48e6e6543a]) successfully deleted
I20240501 20:10:04.251379 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet db31c514cd704786b4be2217daaa11d5 (table rebalance_test_table_16 [id=e07c82123bdc4376bae10c48e6e6543a]) successfully deleted
I20240501 20:10:04.252359 4091 catalog_manager.cc:5576] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205) does not have the latest schema for tablet 0b81e56bfb4c4e93ae68516d9a70538d (table rebalancer_dynamic_table [id=c1beed3afaf146648e40929ee5cc2a29]). Expected version 1 got 0
I20240501 20:10:04.254886 8078 tablet.cc:1722] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:04.256785 4092 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=c1beed3afaf146648e40929ee5cc2a29] alter complete (version 1)
I20240501 20:10:04.258832 8078 tablet.cc:1620] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e: MemRowSet was empty: no flush needed.
I20240501 20:10:04.260154 7872 consensus_queue.cc:1035] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:04.298256 8030 tablet_bootstrap.cc:492] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:04.298754 8030 tablet_bootstrap.cc:492] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap complete.
I20240501 20:10:04.299124 8030 ts_tablet_manager.cc:1367] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.292s user 0.079s sys 0.052s
I20240501 20:10:04.299957 8030 raft_consensus.cc:348] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:04.300237 8030 raft_consensus.cc:729] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: LEARNER
I20240501 20:10:04.300454 8030 consensus_queue.cc:260] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:04.301110 8030 ts_tablet_manager.cc:1398] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:04.303033 4254 tablet_copy_service.cc:342] P dca57823003e4729b769ca12367d7530: Request end of tablet copy session b9957d4086bc4a76b18ed9265a5739d5-34cb2f3c90cc4158a61f426c0e5cb86b received from {username='slave'} at 127.29.219.72:48035
I20240501 20:10:04.303267 4254 tablet_copy_service.cc:433] P dca57823003e4729b769ca12367d7530: ending tablet copy session b9957d4086bc4a76b18ed9265a5739d5-34cb2f3c90cc4158a61f426c0e5cb86b on tablet 34cb2f3c90cc4158a61f426c0e5cb86b with peer b9957d4086bc4a76b18ed9265a5739d5
I20240501 20:10:04.326224 8028 consensus_queue.cc:1035] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:04.342478 7205 consensus_queue.cc:1035] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:04.347987 8038 tablet_bootstrap.cc:492] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:04.348548 8038 tablet_bootstrap.cc:492] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap complete.
I20240501 20:10:04.348877 8038 ts_tablet_manager.cc:1367] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.323s user 0.053s sys 0.011s
I20240501 20:10:04.349857 8038 raft_consensus.cc:348] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:04.350165 8038 raft_consensus.cc:729] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: LEARNER
I20240501 20:10:04.350466 8038 consensus_queue.cc:260] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:04.351328 8038 ts_tablet_manager.cc:1398] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:10:04.354521 4254 tablet_copy_service.cc:342] P dca57823003e4729b769ca12367d7530: Request end of tablet copy session bceeae1c79964acabdae8ff771bf0c97-dfaa3794205d482c90aa4f860fa99cf3 received from {username='slave'} at 127.29.219.71:58089
I20240501 20:10:04.354741 4254 tablet_copy_service.cc:433] P dca57823003e4729b769ca12367d7530: ending tablet copy session bceeae1c79964acabdae8ff771bf0c97-dfaa3794205d482c90aa4f860fa99cf3 on tablet dfaa3794205d482c90aa4f860fa99cf3 with peer bceeae1c79964acabdae8ff771bf0c97
I20240501 20:10:04.359386 8041 tablet_bootstrap.cc:492] T 9d634c1feb7449f686cf3af2a07181e0 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:04.359755 8041 tablet_bootstrap.cc:492] T 9d634c1feb7449f686cf3af2a07181e0 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap complete.
I20240501 20:10:04.359990 8041 ts_tablet_manager.cc:1367] T 9d634c1feb7449f686cf3af2a07181e0 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.377s user 0.050s sys 0.007s
I20240501 20:10:04.360662 8041 raft_consensus.cc:348] T 9d634c1feb7449f686cf3af2a07181e0 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:04.360875 8041 raft_consensus.cc:729] T 9d634c1feb7449f686cf3af2a07181e0 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: LEARNER
I20240501 20:10:04.361063 8041 consensus_queue.cc:260] T 9d634c1feb7449f686cf3af2a07181e0 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:04.361527 8041 ts_tablet_manager.cc:1398] T 9d634c1feb7449f686cf3af2a07181e0 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:10:04.367362 4254 tablet_copy_service.cc:342] P dca57823003e4729b769ca12367d7530: Request end of tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-9d634c1feb7449f686cf3af2a07181e0 received from {username='slave'} at 127.29.219.69:51289
I20240501 20:10:04.367568 4254 tablet_copy_service.cc:433] P dca57823003e4729b769ca12367d7530: ending tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-9d634c1feb7449f686cf3af2a07181e0 on tablet 9d634c1feb7449f686cf3af2a07181e0 with peer ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:04.370182 7879 consensus_queue.cc:1035] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:04.371526 4091 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:04.382572 8074 tablet.cc:1722] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:04.383158 8074 tablet.cc:1620] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:04.383201 8075 tablet.cc:1722] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:04.383633 8075 tablet.cc:1620] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:10:04.383841 8078 tablet.cc:1722] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:04.384377 8078 tablet.cc:1620] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e: MemRowSet was empty: no flush needed.
I20240501 20:10:04.391321 4092 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:04.394870 4087 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=c1beed3afaf146648e40929ee5cc2a29] alter complete (version 2)
I20240501 20:10:04.401513 7400 tablet_service.cc:1467] Processing CreateTablet for tablet d13178f10db0435891b97cebaed6dd5d (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=abd146b01560429fae00683c20d75e9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:04.401968 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d13178f10db0435891b97cebaed6dd5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:04.405544 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 537b5ca846b5485ca1d98351d9038430 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=abd146b01560429fae00683c20d75e9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:04.405997 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 537b5ca846b5485ca1d98351d9038430. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:04.406020 4215 tablet_service.cc:1467] Processing CreateTablet for tablet d13178f10db0435891b97cebaed6dd5d (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=abd146b01560429fae00683c20d75e9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:04.406533 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d13178f10db0435891b97cebaed6dd5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:04.413879 7133 tablet_service.cc:1467] Processing CreateTablet for tablet 537b5ca846b5485ca1d98351d9038430 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=abd146b01560429fae00683c20d75e9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:04.414328 7133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 537b5ca846b5485ca1d98351d9038430. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:04.415130 7000 tablet_service.cc:1467] Processing CreateTablet for tablet c1c51d1b44504379819d96ef5aff1edc (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=abd146b01560429fae00683c20d75e9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:04.415597 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c51d1b44504379819d96ef5aff1edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:04.416149 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 537b5ca846b5485ca1d98351d9038430 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=abd146b01560429fae00683c20d75e9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:04.416672 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 537b5ca846b5485ca1d98351d9038430. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:04.427302 7266 tablet_service.cc:1467] Processing CreateTablet for tablet d13178f10db0435891b97cebaed6dd5d (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=abd146b01560429fae00683c20d75e9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:04.427909 7266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d13178f10db0435891b97cebaed6dd5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:04.431159 7267 tablet_service.cc:1467] Processing CreateTablet for tablet c1c51d1b44504379819d96ef5aff1edc (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=abd146b01560429fae00683c20d75e9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:04.431715 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c51d1b44504379819d96ef5aff1edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:04.433494 8097 tablet_bootstrap.cc:492] T d13178f10db0435891b97cebaed6dd5d P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:04.433066 8100 tablet_bootstrap.cc:492] T 537b5ca846b5485ca1d98351d9038430 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:04.446770 8098 tablet_bootstrap.cc:492] T c1c51d1b44504379819d96ef5aff1edc P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:04.449277 8100 tablet_bootstrap.cc:654] T 537b5ca846b5485ca1d98351d9038430 P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:04.450414 8097 tablet_bootstrap.cc:654] T d13178f10db0435891b97cebaed6dd5d P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:04.452847 8098 tablet_bootstrap.cc:654] T c1c51d1b44504379819d96ef5aff1edc P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:04.455267 7133 tablet_service.cc:1467] Processing CreateTablet for tablet c1c51d1b44504379819d96ef5aff1edc (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=abd146b01560429fae00683c20d75e9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:04.455785 7133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c51d1b44504379819d96ef5aff1edc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:04.457491 8102 tablet_bootstrap.cc:492] T c1c51d1b44504379819d96ef5aff1edc P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:04.459686 8102 tablet_bootstrap.cc:654] T c1c51d1b44504379819d96ef5aff1edc P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:04.461947 8097 tablet_bootstrap.cc:492] T d13178f10db0435891b97cebaed6dd5d P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:04.462184 8097 ts_tablet_manager.cc:1367] T d13178f10db0435891b97cebaed6dd5d P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.000s
I20240501 20:10:04.463088 8097 raft_consensus.cc:348] T d13178f10db0435891b97cebaed6dd5d P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:04.463404 8097 raft_consensus.cc:374] T d13178f10db0435891b97cebaed6dd5d P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:04.463541 8097 raft_consensus.cc:729] T d13178f10db0435891b97cebaed6dd5d P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:04.463879 8097 consensus_queue.cc:260] T d13178f10db0435891b97cebaed6dd5d P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:04.465327 8097 ts_tablet_manager.cc:1398] T d13178f10db0435891b97cebaed6dd5d P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:04.465667 8097 tablet_bootstrap.cc:492] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:04.466660 8100 tablet_bootstrap.cc:492] T 537b5ca846b5485ca1d98351d9038430 P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:04.467897 8106 tablet_bootstrap.cc:492] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:04.468556 8100 ts_tablet_manager.cc:1367] T 537b5ca846b5485ca1d98351d9038430 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.000s
I20240501 20:10:04.471526 8100 raft_consensus.cc:348] T 537b5ca846b5485ca1d98351d9038430 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:04.471776 8100 raft_consensus.cc:374] T 537b5ca846b5485ca1d98351d9038430 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:04.471873 8100 raft_consensus.cc:729] T 537b5ca846b5485ca1d98351d9038430 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:04.471740 8099 tablet_bootstrap.cc:492] T 537b5ca846b5485ca1d98351d9038430 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:04.472126 8100 consensus_queue.cc:260] T 537b5ca846b5485ca1d98351d9038430 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:04.472707 8100 ts_tablet_manager.cc:1398] T 537b5ca846b5485ca1d98351d9038430 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:04.473004 8100 tablet_bootstrap.cc:492] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:04.476068 8106 tablet_bootstrap.cc:654] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:04.476095 8097 tablet_bootstrap.cc:654] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:04.477371 8098 tablet_bootstrap.cc:492] T c1c51d1b44504379819d96ef5aff1edc P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:04.477571 8098 ts_tablet_manager.cc:1367] T c1c51d1b44504379819d96ef5aff1edc P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.000s
I20240501 20:10:04.477805 8099 tablet_bootstrap.cc:654] T 537b5ca846b5485ca1d98351d9038430 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:04.478212 8102 tablet_bootstrap.cc:492] T c1c51d1b44504379819d96ef5aff1edc P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:04.478384 8102 ts_tablet_manager.cc:1367] T c1c51d1b44504379819d96ef5aff1edc P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20240501 20:10:04.478410 8098 raft_consensus.cc:348] T c1c51d1b44504379819d96ef5aff1edc P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:04.478679 8098 raft_consensus.cc:374] T c1c51d1b44504379819d96ef5aff1edc P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:04.478816 8098 raft_consensus.cc:729] T c1c51d1b44504379819d96ef5aff1edc P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:04.479064 8098 consensus_queue.cc:260] T c1c51d1b44504379819d96ef5aff1edc P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:04.479219 8102 raft_consensus.cc:348] T c1c51d1b44504379819d96ef5aff1edc P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:04.479460 8102 raft_consensus.cc:374] T c1c51d1b44504379819d96ef5aff1edc P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:04.479552 8102 raft_consensus.cc:729] T c1c51d1b44504379819d96ef5aff1edc P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:04.481559 8098 ts_tablet_manager.cc:1398] T c1c51d1b44504379819d96ef5aff1edc P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:10:04.483212 8100 tablet_bootstrap.cc:654] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:04.486642 8099 tablet_bootstrap.cc:492] T 537b5ca846b5485ca1d98351d9038430 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:04.486951 8099 ts_tablet_manager.cc:1367] T 537b5ca846b5485ca1d98351d9038430 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20240501 20:10:04.487852 8099 raft_consensus.cc:348] T 537b5ca846b5485ca1d98351d9038430 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:04.488166 8099 raft_consensus.cc:374] T 537b5ca846b5485ca1d98351d9038430 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:04.489364 8106 tablet_bootstrap.cc:492] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:04.489573 8106 ts_tablet_manager.cc:1367] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.006s
I20240501 20:10:04.490505 8106 raft_consensus.cc:348] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:04.490808 8106 raft_consensus.cc:374] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:04.490950 8106 raft_consensus.cc:729] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:04.491868 8106 consensus_queue.cc:260] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:04.492323 8097 tablet_bootstrap.cc:492] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:04.492895 8097 ts_tablet_manager.cc:1367] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20240501 20:10:04.493938 8097 raft_consensus.cc:348] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:04.494833 8102 consensus_queue.cc:260] T c1c51d1b44504379819d96ef5aff1edc P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:04.496351 8106 ts_tablet_manager.cc:1398] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 20:10:04.496534 8099 raft_consensus.cc:729] T 537b5ca846b5485ca1d98351d9038430 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:04.499085 8099 consensus_queue.cc:260] T 537b5ca846b5485ca1d98351d9038430 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:04.500710 8102 ts_tablet_manager.cc:1398] T c1c51d1b44504379819d96ef5aff1edc P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.022s user 0.001s sys 0.002s
I20240501 20:10:04.501057 8097 raft_consensus.cc:374] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:04.501368 8097 raft_consensus.cc:729] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:04.502163 8099 ts_tablet_manager.cc:1398] T 537b5ca846b5485ca1d98351d9038430 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20240501 20:10:04.502776 4091 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:04.503795 8100 tablet_bootstrap.cc:492] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:04.504067 8100 ts_tablet_manager.cc:1367] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.011s
I20240501 20:10:04.505165 8102 tablet_bootstrap.cc:492] T d13178f10db0435891b97cebaed6dd5d P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:04.505451 8097 consensus_queue.cc:260] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:04.506371 8100 raft_consensus.cc:348] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:04.506692 8100 raft_consensus.cc:374] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:04.506815 8100 raft_consensus.cc:729] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:04.507083 8100 consensus_queue.cc:260] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:04.507623 8100 ts_tablet_manager.cc:1398] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:04.508055 8097 ts_tablet_manager.cc:1398] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240501 20:10:04.509524 8102 tablet_bootstrap.cc:654] T d13178f10db0435891b97cebaed6dd5d P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:04.516266 8102 tablet_bootstrap.cc:492] T d13178f10db0435891b97cebaed6dd5d P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:04.516522 8102 ts_tablet_manager.cc:1367] T d13178f10db0435891b97cebaed6dd5d P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.001s
I20240501 20:10:04.517949 8102 raft_consensus.cc:348] T d13178f10db0435891b97cebaed6dd5d P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:04.518262 8102 raft_consensus.cc:374] T d13178f10db0435891b97cebaed6dd5d P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:04.518397 8102 raft_consensus.cc:729] T d13178f10db0435891b97cebaed6dd5d P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:04.518719 8102 consensus_queue.cc:260] T d13178f10db0435891b97cebaed6dd5d P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:04.519389 8102 ts_tablet_manager.cc:1398] T d13178f10db0435891b97cebaed6dd5d P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:04.522315 8074 tablet.cc:1722] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:04.522818 8075 tablet.cc:1722] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:04.527961 8078 tablet.cc:1722] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:04.528658 8093 raft_consensus.cc:482] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:04.528832 8093 raft_consensus.cc:504] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:04.529560 8093 leader_election.cc:288] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:04.530319 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d13178f10db0435891b97cebaed6dd5d" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:04.530570 7284 raft_consensus.cc:2454] T d13178f10db0435891b97cebaed6dd5d P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:04.531155 4168 leader_election.cc:302] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:04.531450 8093 raft_consensus.cc:2781] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:04.531565 8093 raft_consensus.cc:482] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:04.531651 8093 raft_consensus.cc:3037] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.532523 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d13178f10db0435891b97cebaed6dd5d" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:04.532759 7420 raft_consensus.cc:2454] T d13178f10db0435891b97cebaed6dd5d P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:04.536836 8093 raft_consensus.cc:504] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:04.538214 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d13178f10db0435891b97cebaed6dd5d" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:04.538352 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d13178f10db0435891b97cebaed6dd5d" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:04.538453 7284 raft_consensus.cc:3037] T d13178f10db0435891b97cebaed6dd5d P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.538556 7420 raft_consensus.cc:3037] T d13178f10db0435891b97cebaed6dd5d P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.541378 7420 raft_consensus.cc:2454] T d13178f10db0435891b97cebaed6dd5d P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:04.541378 7284 raft_consensus.cc:2454] T d13178f10db0435891b97cebaed6dd5d P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:04.541854 4170 leader_election.cc:302] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:04.541983 4087 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table_new_name [id=c1beed3afaf146648e40929ee5cc2a29] alter complete (version 3)
I20240501 20:10:04.542145 7205 raft_consensus.cc:2781] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:04.542296 7205 raft_consensus.cc:686] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:10:04.542559 7205 consensus_queue.cc:237] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:04.543432 8093 leader_election.cc:288] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:04.549234 4090 catalog_manager.cc:5484] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:04.588408 7420 raft_consensus.cc:1203] T c31148e9e4dd42c3a11787d9384c79c2 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:04.593890 4084 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240501 20:10:04.594143 4084 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240501 20:10:04.602344 4084 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 80d18632bc0443558294b9f99bf08654
I20240501 20:10:04.603345 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d18632bc0443558294b9f99bf08654 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:04 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:04.603629 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d18632bc0443558294b9f99bf08654 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:04 UTC) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:04.603922 7512 tablet_replica.cc:330] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:04.603953 7519 tablet_replica.cc:330] T 80d18632bc0443558294b9f99bf08654 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:04.604158 7519 raft_consensus.cc:2229] T 80d18632bc0443558294b9f99bf08654 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:04.604147 7512 raft_consensus.cc:2229] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:04.604369 7519 raft_consensus.cc:2258] T 80d18632bc0443558294b9f99bf08654 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:04.604511 7512 raft_consensus.cc:2258] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:04.605111 7519 ts_tablet_manager.cc:1875] T 80d18632bc0443558294b9f99bf08654 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:04.605221 7512 ts_tablet_manager.cc:1875] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:04.607347 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d18632bc0443558294b9f99bf08654 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:04 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:04.607586 7722 tablet_replica.cc:330] T 80d18632bc0443558294b9f99bf08654 P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:04.607775 7722 raft_consensus.cc:2229] T 80d18632bc0443558294b9f99bf08654 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:04.607983 7722 raft_consensus.cc:2258] T 80d18632bc0443558294b9f99bf08654 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:04.608650 7722 ts_tablet_manager.cc:1875] T 80d18632bc0443558294b9f99bf08654 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:04.603682 4084 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet c2c09fbc593b4fb5a97e05f5c8964522
I20240501 20:10:04.610498 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet c2c09fbc593b4fb5a97e05f5c8964522 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:04 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:04.611112 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet c2c09fbc593b4fb5a97e05f5c8964522 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:04 UTC) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:04.611404 7847 tablet_replica.cc:330] T c2c09fbc593b4fb5a97e05f5c8964522 P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:04.611621 7847 raft_consensus.cc:2229] T c2c09fbc593b4fb5a97e05f5c8964522 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:04.611881 7847 raft_consensus.cc:2258] T c2c09fbc593b4fb5a97e05f5c8964522 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:04.612625 7847 ts_tablet_manager.cc:1875] T c2c09fbc593b4fb5a97e05f5c8964522 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:04.613176 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet c2c09fbc593b4fb5a97e05f5c8964522 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:04 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:04.618157 7847 ts_tablet_manager.cc:1888] T c2c09fbc593b4fb5a97e05f5c8964522 P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:04.619004 7847 log.cc:1196] T c2c09fbc593b4fb5a97e05f5c8964522 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/c2c09fbc593b4fb5a97e05f5c8964522
I20240501 20:10:04.619572 7847 ts_tablet_manager.cc:1909] T c2c09fbc593b4fb5a97e05f5c8964522 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting consensus metadata
I20240501 20:10:04.620656 4084 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet cde1b2be0bcd4337baed241933703f42
I20240501 20:10:04.620767 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet c2c09fbc593b4fb5a97e05f5c8964522 (table rebalance_test_table_18 [id=b5dc7891d8a1410bb1280ac3ce2fd364]) successfully deleted
I20240501 20:10:04.622001 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet cde1b2be0bcd4337baed241933703f42 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:04 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:04.622577 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet cde1b2be0bcd4337baed241933703f42 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:04 UTC) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:04.622829 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet cde1b2be0bcd4337baed241933703f42 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:04 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:04.624898 8028 raft_consensus.cc:482] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:04.625003 7512 ts_tablet_manager.cc:1888] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:04.625169 7512 log.cc:1196] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/80d18632bc0443558294b9f99bf08654
I20240501 20:10:04.625088 8028 raft_consensus.cc:504] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:04.625584 7512 ts_tablet_manager.cc:1909] T 80d18632bc0443558294b9f99bf08654 P 10b47782558c41ee91f5e164a372049a: Deleting consensus metadata
I20240501 20:10:04.625963 8028 leader_election.cc:288] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:04.627725 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537b5ca846b5485ca1d98351d9038430" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:04.627990 7155 raft_consensus.cc:2454] T 537b5ca846b5485ca1d98351d9038430 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:10:04.628571 7353 leader_election.cc:302] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:04.631899 7722 ts_tablet_manager.cc:1888] T 80d18632bc0443558294b9f99bf08654 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:04.632100 7722 log.cc:1196] T 80d18632bc0443558294b9f99bf08654 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/80d18632bc0443558294b9f99bf08654
I20240501 20:10:04.632691 7722 ts_tablet_manager.cc:1909] T 80d18632bc0443558294b9f99bf08654 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:04.633916 7722 tablet_replica.cc:330] T c2c09fbc593b4fb5a97e05f5c8964522 P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:04.634189 7722 raft_consensus.cc:2229] T c2c09fbc593b4fb5a97e05f5c8964522 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:04.634177 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet 80d18632bc0443558294b9f99bf08654 (table rebalance_test_table_18 [id=b5dc7891d8a1410bb1280ac3ce2fd364]) successfully deleted
I20240501 20:10:04.634454 7722 raft_consensus.cc:2258] T c2c09fbc593b4fb5a97e05f5c8964522 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:04.635438 8028 raft_consensus.cc:2781] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:04.635550 8028 raft_consensus.cc:482] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:04.635653 8028 raft_consensus.cc:3037] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.645929 7512 tablet_replica.cc:330] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:04.646234 7512 raft_consensus.cc:2229] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:04.646684 7512 raft_consensus.cc:2258] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:04.646757 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet 80d18632bc0443558294b9f99bf08654 (table rebalance_test_table_18 [id=b5dc7891d8a1410bb1280ac3ce2fd364]) successfully deleted
I20240501 20:10:04.647902 7512 ts_tablet_manager.cc:1875] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:04.649634 4091 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:04.649925 4091 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:04.651451 7519 ts_tablet_manager.cc:1888] T 80d18632bc0443558294b9f99bf08654 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:04.651609 7519 log.cc:1196] T 80d18632bc0443558294b9f99bf08654 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/80d18632bc0443558294b9f99bf08654
I20240501 20:10:04.652004 7519 ts_tablet_manager.cc:1909] T 80d18632bc0443558294b9f99bf08654 P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:04.653095 7519 tablet_replica.cc:330] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:04.653363 7519 raft_consensus.cc:2229] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:04.653765 7519 raft_consensus.cc:2258] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:04.654703 7519 ts_tablet_manager.cc:1875] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:04.654911 7512 ts_tablet_manager.cc:1888] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:04.655136 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet 80d18632bc0443558294b9f99bf08654 (table rebalance_test_table_18 [id=b5dc7891d8a1410bb1280ac3ce2fd364]) successfully deleted
I20240501 20:10:04.655126 7512 log.cc:1196] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/cde1b2be0bcd4337baed241933703f42
I20240501 20:10:04.655648 7722 ts_tablet_manager.cc:1875] T c2c09fbc593b4fb5a97e05f5c8964522 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:04.655925 7512 ts_tablet_manager.cc:1909] T cde1b2be0bcd4337baed241933703f42 P 10b47782558c41ee91f5e164a372049a: Deleting consensus metadata
I20240501 20:10:04.657351 4091 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 0b81e56bfb4c4e93ae68516d9a70538d
I20240501 20:10:04.657863 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet cde1b2be0bcd4337baed241933703f42 (table rebalance_test_table_18 [id=b5dc7891d8a1410bb1280ac3ce2fd364]) successfully deleted
I20240501 20:10:04.658519 7133 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b81e56bfb4c4e93ae68516d9a70538d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:04 UTC) from {username='slave'} at 127.0.0.1:52036
I20240501 20:10:04.658703 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b81e56bfb4c4e93ae68516d9a70538d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:04 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:04.658859 8060 tablet_replica.cc:330] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd: stopping tablet replica
I20240501 20:10:04.659086 8060 raft_consensus.cc:2229] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:04.659293 8060 raft_consensus.cc:2258] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:04.660357 8060 ts_tablet_manager.cc:1875] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:04.661816 7519 ts_tablet_manager.cc:1888] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:04.661959 7519 log.cc:1196] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/c2c09fbc593b4fb5a97e05f5c8964522
I20240501 20:10:04.662226 8028 raft_consensus.cc:504] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:04.662415 7519 ts_tablet_manager.cc:1909] T c2c09fbc593b4fb5a97e05f5c8964522 P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:04.663267 7000 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b81e56bfb4c4e93ae68516d9a70538d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:04 UTC) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:04.663386 8028 leader_election.cc:288] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:04.663911 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet c2c09fbc593b4fb5a97e05f5c8964522 (table rebalance_test_table_18 [id=b5dc7891d8a1410bb1280ac3ce2fd364]) successfully deleted
I20240501 20:10:04.664265 7519 tablet_replica.cc:330] T cde1b2be0bcd4337baed241933703f42 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:04.664530 7519 raft_consensus.cc:2229] T cde1b2be0bcd4337baed241933703f42 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:04.664547 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537b5ca846b5485ca1d98351d9038430" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:04.664778 7155 raft_consensus.cc:3037] T 537b5ca846b5485ca1d98351d9038430 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.664810 7519 raft_consensus.cc:2258] T cde1b2be0bcd4337baed241933703f42 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:04.665518 7519 ts_tablet_manager.cc:1875] T cde1b2be0bcd4337baed241933703f42 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:04.668514 7155 raft_consensus.cc:2454] T 537b5ca846b5485ca1d98351d9038430 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:04.669085 7353 leader_election.cc:302] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:04.669382 8028 raft_consensus.cc:2781] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:04.669607 8028 raft_consensus.cc:686] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Becoming Leader. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Running, Role: LEADER
I20240501 20:10:04.669904 8028 consensus_queue.cc:237] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:04.672648 8060 ts_tablet_manager.cc:1888] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:04.672833 7519 ts_tablet_manager.cc:1888] T cde1b2be0bcd4337baed241933703f42 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:04.673059 7519 log.cc:1196] T cde1b2be0bcd4337baed241933703f42 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/cde1b2be0bcd4337baed241933703f42
I20240501 20:10:04.673564 7519 ts_tablet_manager.cc:1909] T cde1b2be0bcd4337baed241933703f42 P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:04.673909 8060 log.cc:1196] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/wals/0b81e56bfb4c4e93ae68516d9a70538d
I20240501 20:10:04.675050 8060 ts_tablet_manager.cc:1909] T 0b81e56bfb4c4e93ae68516d9a70538d P db0742ee3860468590eefef6ae6074fd: Deleting consensus metadata
I20240501 20:10:04.675233 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet cde1b2be0bcd4337baed241933703f42 (table rebalance_test_table_18 [id=b5dc7891d8a1410bb1280ac3ce2fd364]) successfully deleted
I20240501 20:10:04.678290 4076 catalog_manager.cc:4830] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): tablet 0b81e56bfb4c4e93ae68516d9a70538d (table rebalancer_dynamic_table_new_name [id=c1beed3afaf146648e40929ee5cc2a29]) successfully deleted
I20240501 20:10:04.683708 8136 tablet_replica.cc:330] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d: stopping tablet replica
I20240501 20:10:04.684053 8136 raft_consensus.cc:2229] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:04.684480 8136 raft_consensus.cc:2258] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:04.685446 8136 ts_tablet_manager.cc:1875] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:04.685460 4091 catalog_manager.cc:5484] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72). New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:04.690554 7722 ts_tablet_manager.cc:1888] T c2c09fbc593b4fb5a97e05f5c8964522 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:04.690860 7722 log.cc:1196] T c2c09fbc593b4fb5a97e05f5c8964522 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/c2c09fbc593b4fb5a97e05f5c8964522
I20240501 20:10:04.691448 7722 ts_tablet_manager.cc:1909] T c2c09fbc593b4fb5a97e05f5c8964522 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:04.692559 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet c2c09fbc593b4fb5a97e05f5c8964522 (table rebalance_test_table_18 [id=b5dc7891d8a1410bb1280ac3ce2fd364]) successfully deleted
I20240501 20:10:04.692963 7722 tablet_replica.cc:330] T cde1b2be0bcd4337baed241933703f42 P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:04.693226 7722 raft_consensus.cc:2229] T cde1b2be0bcd4337baed241933703f42 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:04.693491 7722 raft_consensus.cc:2258] T cde1b2be0bcd4337baed241933703f42 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:04.694355 7722 ts_tablet_manager.cc:1875] T cde1b2be0bcd4337baed241933703f42 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:04.702113 7850 raft_consensus.cc:482] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:04.702309 7850 raft_consensus.cc:504] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:04.702996 7850 leader_election.cc:288] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:04.704170 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c51d1b44504379819d96ef5aff1edc" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:04.704423 7022 raft_consensus.cc:2454] T c1c51d1b44504379819d96ef5aff1edc P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:04.705240 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c51d1b44504379819d96ef5aff1edc" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:04.705471 7284 raft_consensus.cc:2454] T c1c51d1b44504379819d96ef5aff1edc P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:04.706394 7089 leader_election.cc:302] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:04.707051 8136 ts_tablet_manager.cc:1888] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:04.707255 8136 log.cc:1196] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/wals/0b81e56bfb4c4e93ae68516d9a70538d
I20240501 20:10:04.707883 8136 ts_tablet_manager.cc:1909] T 0b81e56bfb4c4e93ae68516d9a70538d P ef0a4ada548446b3a531cb39a3c4e38d: Deleting consensus metadata
I20240501 20:10:04.708511 7850 raft_consensus.cc:2781] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:04.708621 7850 raft_consensus.cc:482] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:04.708753 7850 raft_consensus.cc:3037] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.712917 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet 0b81e56bfb4c4e93ae68516d9a70538d (table rebalancer_dynamic_table_new_name [id=c1beed3afaf146648e40929ee5cc2a29]) successfully deleted
I20240501 20:10:04.714318 7850 raft_consensus.cc:504] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:04.716742 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c51d1b44504379819d96ef5aff1edc" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:04.717000 7022 raft_consensus.cc:3037] T c1c51d1b44504379819d96ef5aff1edc P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.719977 7022 raft_consensus.cc:2454] T c1c51d1b44504379819d96ef5aff1edc P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:04.725759 7090 leader_election.cc:302] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db0742ee3860468590eefef6ae6074fd, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:04.737218 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c51d1b44504379819d96ef5aff1edc" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:04.737499 7284 raft_consensus.cc:3037] T c1c51d1b44504379819d96ef5aff1edc P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.740185 7284 raft_consensus.cc:2454] T c1c51d1b44504379819d96ef5aff1edc P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:04.741322 7850 leader_election.cc:288] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:04.741724 7850 raft_consensus.cc:2781] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:04.741874 7850 raft_consensus.cc:686] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Becoming Leader. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Running, Role: LEADER
I20240501 20:10:04.742133 7850 consensus_queue.cc:237] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:04.747601 4091 catalog_manager.cc:5484] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd reported cstate change: term changed from 0 to 1, leader changed from <none> to db0742ee3860468590eefef6ae6074fd (127.29.219.70). New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:04.757100 7932 raft_consensus.cc:1050] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b: attempting to promote NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d to VOTER
I20240501 20:10:04.758112 7932 consensus_queue.cc:237] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:04.760545 4091 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:04.764103 4367 raft_consensus.cc:1261] T 46df0faa681b469abe9e383b181a6053 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:04.766517 6775 consensus_queue.cc:1035] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:04.770404 7020 raft_consensus.cc:1261] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:04.773275 7420 raft_consensus.cc:1203] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:04.773818 7919 consensus_queue.cc:1035] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:04.774839 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537b5ca846b5485ca1d98351d9038430" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:04.775184 4631 raft_consensus.cc:2454] T 537b5ca846b5485ca1d98351d9038430 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:10:04.775552 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "537b5ca846b5485ca1d98351d9038430" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:10:04.775882 4630 raft_consensus.cc:3037] T 537b5ca846b5485ca1d98351d9038430 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.781915 7267 tablet_service.cc:1467] Processing CreateTablet for tablet 040734285a8347408f515daf91b43a55 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c5969daedc4945f385966e8e481f3e7a]), partition=
I20240501 20:10:04.782353 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 040734285a8347408f515daf91b43a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:04.782642 7000 tablet_service.cc:1467] Processing CreateTablet for tablet 040734285a8347408f515daf91b43a55 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c5969daedc4945f385966e8e481f3e7a]), partition=
I20240501 20:10:04.783012 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 040734285a8347408f515daf91b43a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:04.785895 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 040734285a8347408f515daf91b43a55 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c5969daedc4945f385966e8e481f3e7a]), partition=
I20240501 20:10:04.786289 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 040734285a8347408f515daf91b43a55. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 20:10:04.786713 7090 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.219.69:46237, user_credentials={real_user=slave}} blocked reactor thread for 60983.3us
I20240501 20:10:04.787254 4630 raft_consensus.cc:2454] T 537b5ca846b5485ca1d98351d9038430 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:04.790644 8098 tablet_bootstrap.cc:492] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:04.791262 4630 raft_consensus.cc:1261] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:04.792558 8098 tablet_bootstrap.cc:654] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:04.793953 8102 tablet_bootstrap.cc:492] T 040734285a8347408f515daf91b43a55 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:04.795727 8102 tablet_bootstrap.cc:654] T 040734285a8347408f515daf91b43a55 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:04.799945 7284 raft_consensus.cc:1203] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:04.802546 8097 tablet_bootstrap.cc:492] T 040734285a8347408f515daf91b43a55 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:04.803433 7919 consensus_queue.cc:1035] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:04.804806 8097 tablet_bootstrap.cc:654] T 040734285a8347408f515daf91b43a55 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:04.813632 7722 ts_tablet_manager.cc:1888] T cde1b2be0bcd4337baed241933703f42 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 20:10:04.814206 7722 log.cc:1196] T cde1b2be0bcd4337baed241933703f42 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/cde1b2be0bcd4337baed241933703f42
I20240501 20:10:04.815261 7722 ts_tablet_manager.cc:1909] T cde1b2be0bcd4337baed241933703f42 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:04.818078 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet cde1b2be0bcd4337baed241933703f42 (table rebalance_test_table_18 [id=b5dc7891d8a1410bb1280ac3ce2fd364]) successfully deleted
I20240501 20:10:04.820405 8097 tablet_bootstrap.cc:492] T 040734285a8347408f515daf91b43a55 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:04.820626 8097 ts_tablet_manager.cc:1367] T 040734285a8347408f515daf91b43a55 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.005s
I20240501 20:10:04.821779 8097 raft_consensus.cc:348] T 040734285a8347408f515daf91b43a55 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:04.822187 8097 raft_consensus.cc:374] T 040734285a8347408f515daf91b43a55 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:04.822340 8097 raft_consensus.cc:729] T 040734285a8347408f515daf91b43a55 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:04.822721 8097 consensus_queue.cc:260] T 040734285a8347408f515daf91b43a55 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:04.823907 7932 raft_consensus.cc:2932] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:04.824534 8097 ts_tablet_manager.cc:1398] T 040734285a8347408f515daf91b43a55 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 20:10:04.825171 8102 tablet_bootstrap.cc:492] T 040734285a8347408f515daf91b43a55 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:04.825323 8102 ts_tablet_manager.cc:1367] T 040734285a8347408f515daf91b43a55 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.008s
I20240501 20:10:04.829294 7722 tablet_replica.cc:330] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:04.827515 8102 raft_consensus.cc:348] T 040734285a8347408f515daf91b43a55 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:04.829787 7722 raft_consensus.cc:2229] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:04.829977 8102 raft_consensus.cc:374] T 040734285a8347408f515daf91b43a55 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:04.830132 8102 raft_consensus.cc:729] T 040734285a8347408f515daf91b43a55 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:04.830399 7722 raft_consensus.cc:2258] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:04.831815 4084 catalog_manager.cc:5484] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:04.832221 7022 raft_consensus.cc:2932] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:04.834244 8098 tablet_bootstrap.cc:492] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:04.834494 8098 ts_tablet_manager.cc:1367] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.008s
I20240501 20:10:04.835440 8098 raft_consensus.cc:348] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:04.835800 8098 raft_consensus.cc:374] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:04.835944 8098 raft_consensus.cc:729] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:04.836292 8098 consensus_queue.cc:260] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:04.837286 4497 consensus_queue.cc:237] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:04.845356 8102 consensus_queue.cc:260] T 040734285a8347408f515daf91b43a55 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:04.846045 8102 ts_tablet_manager.cc:1398] T 040734285a8347408f515daf91b43a55 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20240501 20:10:04.838135 4630 raft_consensus.cc:2932] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:04.843019 8098 ts_tablet_manager.cc:1398] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 20:10:04.843761 7022 raft_consensus.cc:1261] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:04.850988 8153 consensus_queue.cc:1035] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:04.853502 7722 ts_tablet_manager.cc:1875] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:04.854120 6775 raft_consensus.cc:2932] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:04.859174 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 46df0faa681b469abe9e383b181a6053 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:04.860414 4631 raft_consensus.cc:1261] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:04.860307 7022 raft_consensus.cc:2932] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:04.863554 7919 consensus_queue.cc:1035] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:04.864552 7020 raft_consensus.cc:1203] T 9d634c1feb7449f686cf3af2a07181e0 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:04.869493 7722 ts_tablet_manager.cc:1888] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:04.872917 4092 catalog_manager.cc:5484] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:04.872948 7722 log.cc:1196] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/0b81e56bfb4c4e93ae68516d9a70538d
I20240501 20:10:04.875927 7722 ts_tablet_manager.cc:1909] T 0b81e56bfb4c4e93ae68516d9a70538d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:04.881287 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet 46df0faa681b469abe9e383b181a6053 with delete_type TABLET_DATA_TOMBSTONED (TS 10b47782558c41ee91f5e164a372049a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:04.881775 7512 tablet_replica.cc:330] T 46df0faa681b469abe9e383b181a6053 P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:04.881996 7512 raft_consensus.cc:2229] T 46df0faa681b469abe9e383b181a6053 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:04.882369 7512 pending_rounds.cc:62] T 46df0faa681b469abe9e383b181a6053 P 10b47782558c41ee91f5e164a372049a: Trying to abort 1 pending ops.
I20240501 20:10:04.882506 7512 pending_rounds.cc:68] T 46df0faa681b469abe9e383b181a6053 P 10b47782558c41ee91f5e164a372049a: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022977862686982144 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "46df0faa681b469abe9e383b181a6053" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } } }
I20240501 20:10:04.883185 7512 raft_consensus.cc:2894] T 46df0faa681b469abe9e383b181a6053 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 20:10:04.883313 7512 raft_consensus.cc:2866] T 46df0faa681b469abe9e383b181a6053 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:10:04.883440 7512 raft_consensus.cc:2258] T 46df0faa681b469abe9e383b181a6053 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:04.884060 8019 raft_consensus.cc:482] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:04.884258 8019 raft_consensus.cc:504] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:04.885180 8019 leader_election.cc:288] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:04.886122 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "040734285a8347408f515daf91b43a55" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:04.886368 7284 raft_consensus.cc:2454] T 040734285a8347408f515daf91b43a55 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:04.886883 6956 leader_election.cc:302] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:04.887077 7512 ts_tablet_manager.cc:1875] T 46df0faa681b469abe9e383b181a6053 P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:04.887430 8019 raft_consensus.cc:2781] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:04.887547 8019 raft_consensus.cc:482] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:04.887639 8019 raft_consensus.cc:3037] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.891767 8019 raft_consensus.cc:504] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:04.894990 7512 ts_tablet_manager.cc:1888] T 46df0faa681b469abe9e383b181a6053 P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:04.877360 4631 raft_consensus.cc:2932] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:04.897248 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "040734285a8347408f515daf91b43a55" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:04.897549 7284 raft_consensus.cc:3037] T 040734285a8347408f515daf91b43a55 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.899864 7512 log.cc:1196] T 46df0faa681b469abe9e383b181a6053 P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/46df0faa681b469abe9e383b181a6053
I20240501 20:10:04.900806 7284 raft_consensus.cc:2454] T 040734285a8347408f515daf91b43a55 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:04.900939 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet 46df0faa681b469abe9e383b181a6053 (table rebalance_test_table_13 [id=e21b0047cbf842c6833a78db81a34171]) successfully deleted
I20240501 20:10:04.901445 6956 leader_election.cc:302] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:04.901829 7880 raft_consensus.cc:2781] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:04.901984 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet 0b81e56bfb4c4e93ae68516d9a70538d (table rebalancer_dynamic_table_new_name [id=c1beed3afaf146648e40929ee5cc2a29]) successfully deleted
I20240501 20:10:04.903043 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "040734285a8347408f515daf91b43a55" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:04.903312 7420 raft_consensus.cc:2454] T 040734285a8347408f515daf91b43a55 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:04.914348 8019 leader_election.cc:288] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Requested vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:04.915169 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "040734285a8347408f515daf91b43a55" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:04.915395 7420 raft_consensus.cc:3037] T 040734285a8347408f515daf91b43a55 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:04.918467 7420 raft_consensus.cc:2454] T 040734285a8347408f515daf91b43a55 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:04.950392 4347 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:10:04.951411 7880 raft_consensus.cc:686] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Becoming Leader. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Running, Role: LEADER
I20240501 20:10:04.951871 7880 consensus_queue.cc:237] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:04.959932 4091 catalog_manager.cc:5484] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69). New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:04.968346 8133 raft_consensus.cc:1050] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a: attempting to promote NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 to VOTER
I20240501 20:10:04.968931 7133 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 20:10:04.969040 7401 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:04.969470 8133 consensus_queue.cc:237] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:04.980352 4215 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:04.983603 4610 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:10:05.008807 7000 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:05.009418 4478 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:05.011298 7420 raft_consensus.cc:1261] T c31148e9e4dd42c3a11787d9384c79c2 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:05.011413 4627 raft_consensus.cc:1261] T c31148e9e4dd42c3a11787d9384c79c2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:05.011921 4497 raft_consensus.cc:1261] T c31148e9e4dd42c3a11787d9384c79c2 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:05.016139 8093 consensus_queue.cc:1035] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:05.031773 8093 consensus_queue.cc:1035] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:05.040865 7267 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:10:05.044858 7854 consensus_queue.cc:1035] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20240501 20:10:05.046840 8133 consensus_queue.cc:1035] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20240501 20:10:05.051914 7854 consensus_queue.cc:1035] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:05.084481 8177 raft_consensus.cc:2932] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:05.095873 4091 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:05.098502 4497 raft_consensus.cc:2932] T c31148e9e4dd42c3a11787d9384c79c2 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:05.097684 7419 raft_consensus.cc:2932] T c31148e9e4dd42c3a11787d9384c79c2 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:05.101609 4084 catalog_manager.cc:5484] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:05.119357 4367 consensus_queue.cc:237] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:05.122238 4630 raft_consensus.cc:1261] T c31148e9e4dd42c3a11787d9384c79c2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:05.125820 8175 consensus_queue.cc:1035] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:05.126435 4627 raft_consensus.cc:2932] T c31148e9e4dd42c3a11787d9384c79c2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:05.134442 7419 raft_consensus.cc:1261] T c31148e9e4dd42c3a11787d9384c79c2 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:05.135166 8175 consensus_queue.cc:1035] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:05.137969 8175 raft_consensus.cc:2932] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:05.143031 7421 raft_consensus.cc:2932] T c31148e9e4dd42c3a11787d9384c79c2 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:05.149804 7287 raft_consensus.cc:1261] T 040734285a8347408f515daf91b43a55 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:05.151175 7880 consensus_queue.cc:1035] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:05.152334 7420 raft_consensus.cc:1261] T 040734285a8347408f515daf91b43a55 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:05.153934 7880 consensus_queue.cc:1035] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:05.164609 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c31148e9e4dd42c3a11787d9384c79c2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:05.165702 4630 raft_consensus.cc:2932] T c31148e9e4dd42c3a11787d9384c79c2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:05.169361 4091 catalog_manager.cc:5484] T c31148e9e4dd42c3a11787d9384c79c2 P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:05.181696 7850 consensus_queue.cc:1035] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:05.187038 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet c31148e9e4dd42c3a11787d9384c79c2 with delete_type TABLET_DATA_TOMBSTONED (TS 3e974c03d3c546f9ab6c91e762a5426b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:05.207392 8187 tablet_replica.cc:330] T c31148e9e4dd42c3a11787d9384c79c2 P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:05.207662 8187 raft_consensus.cc:2229] T c31148e9e4dd42c3a11787d9384c79c2 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:05.208053 8187 raft_consensus.cc:2258] T c31148e9e4dd42c3a11787d9384c79c2 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:05.212877 4092 catalog_manager.cc:5576] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447) does not have the latest schema for tablet 040734285a8347408f515daf91b43a55 (table rebalancer_dynamic_table [id=c5969daedc4945f385966e8e481f3e7a]). Expected version 1 got 0
I20240501 20:10:05.217361 8028 consensus_queue.cc:1035] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:05.231678 7872 consensus_queue.cc:1035] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:05.243866 7657 raft_consensus.cc:1050] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530: attempting to promote NON_VOTER bceeae1c79964acabdae8ff771bf0c97 to VOTER
I20240501 20:10:05.244927 7657 consensus_queue.cc:237] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:05.250461 8187 ts_tablet_manager.cc:1875] T c31148e9e4dd42c3a11787d9384c79c2 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:05.253310 8192 tablet.cc:1722] T 040734285a8347408f515daf91b43a55 P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:05.261216 4092 catalog_manager.cc:5576] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447) does not have the latest schema for tablet 040734285a8347408f515daf91b43a55 (table rebalancer_dynamic_table [id=c5969daedc4945f385966e8e481f3e7a]). Expected version 1 got 0
I20240501 20:10:05.264806 8139 consensus_queue.cc:1035] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:05.265259 8188 tablet.cc:1722] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:05.265998 8188 tablet.cc:1620] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:05.267242 4084 catalog_manager.cc:5576] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817) does not have the latest schema for tablet 040734285a8347408f515daf91b43a55 (table rebalancer_dynamic_table [id=c5969daedc4945f385966e8e481f3e7a]). Expected version 1 got 0
I20240501 20:10:05.267907 8192 tablet.cc:1620] T 040734285a8347408f515daf91b43a55 P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:05.275331 4084 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=c5969daedc4945f385966e8e481f3e7a] alter complete (version 1)
I20240501 20:10:05.288234 4091 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:05.292876 7284 raft_consensus.cc:1261] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:05.293627 7205 consensus_queue.cc:1035] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:05.299129 4497 raft_consensus.cc:1261] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:05.299791 7205 consensus_queue.cc:1035] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:05.301964 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 3f63c61d85524d87a4b911c321f486c4 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=817a9d24b5a445c4be8c44bcb31c8e41]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:05.302448 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f63c61d85524d87a4b911c321f486c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.302465 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 72ddbf0c6a4e4c1fb3c8d59288894a35 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=817a9d24b5a445c4be8c44bcb31c8e41]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:05.302995 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ddbf0c6a4e4c1fb3c8d59288894a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.303752 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 72ddbf0c6a4e4c1fb3c8d59288894a35 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=817a9d24b5a445c4be8c44bcb31c8e41]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:05.304183 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ddbf0c6a4e4c1fb3c8d59288894a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.307335 4347 tablet_service.cc:1467] Processing CreateTablet for tablet 6e3d6a53365f4eceac1d99d4cd51153b (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=817a9d24b5a445c4be8c44bcb31c8e41]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:05.307796 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e3d6a53365f4eceac1d99d4cd51153b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.307904 7133 tablet_service.cc:1467] Processing CreateTablet for tablet 72ddbf0c6a4e4c1fb3c8d59288894a35 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=817a9d24b5a445c4be8c44bcb31c8e41]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:05.308305 7133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72ddbf0c6a4e4c1fb3c8d59288894a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.311103 7000 tablet_service.cc:1467] Processing CreateTablet for tablet 3f63c61d85524d87a4b911c321f486c4 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=817a9d24b5a445c4be8c44bcb31c8e41]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:05.311553 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f63c61d85524d87a4b911c321f486c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.320745 7267 tablet_service.cc:1467] Processing CreateTablet for tablet 3f63c61d85524d87a4b911c321f486c4 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=817a9d24b5a445c4be8c44bcb31c8e41]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:05.321292 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f63c61d85524d87a4b911c321f486c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.324052 8098 tablet_bootstrap.cc:492] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:05.325791 8098 tablet_bootstrap.cc:654] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:05.326260 4630 raft_consensus.cc:1261] T dfaa3794205d482c90aa4f860fa99cf3 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:05.327000 8093 consensus_queue.cc:1035] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:05.336689 7134 tablet_service.cc:1467] Processing CreateTablet for tablet 6e3d6a53365f4eceac1d99d4cd51153b (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=817a9d24b5a445c4be8c44bcb31c8e41]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:05.337261 7134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e3d6a53365f4eceac1d99d4cd51153b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.348591 7266 tablet_service.cc:1467] Processing CreateTablet for tablet 6e3d6a53365f4eceac1d99d4cd51153b (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=817a9d24b5a445c4be8c44bcb31c8e41]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:05.349114 7266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e3d6a53365f4eceac1d99d4cd51153b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.357234 8206 tablet_bootstrap.cc:492] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:05.357515 8102 tablet_bootstrap.cc:492] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:05.359280 8206 tablet_bootstrap.cc:654] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:05.359273 8102 tablet_bootstrap.cc:654] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:05.359728 8093 raft_consensus.cc:2932] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:05.362143 8201 tablet_bootstrap.cc:492] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:05.366752 8201 tablet_bootstrap.cc:654] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:05.367532 8203 tablet_bootstrap.cc:492] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:05.367882 4498 raft_consensus.cc:2932] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:05.370798 8203 tablet_bootstrap.cc:654] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:05.371382 8202 tablet_bootstrap.cc:492] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:05.375380 4092 catalog_manager.cc:5484] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:05.374171 8200 tablet.cc:1722] T 040734285a8347408f515daf91b43a55 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:05.379611 4091 catalog_manager.cc:5576] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817) does not have the latest schema for tablet 040734285a8347408f515daf91b43a55 (table rebalancer_dynamic_table [id=c5969daedc4945f385966e8e481f3e7a]). Expected version 1 got 0
I20240501 20:10:05.381582 8102 tablet_bootstrap.cc:492] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:05.382941 8102 ts_tablet_manager.cc:1367] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.003s
I20240501 20:10:05.383782 8102 raft_consensus.cc:348] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:05.384212 8102 raft_consensus.cc:374] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:05.384344 8102 raft_consensus.cc:729] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.384711 8102 consensus_queue.cc:260] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:05.385444 8187 ts_tablet_manager.cc:1888] T c31148e9e4dd42c3a11787d9384c79c2 P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:05.386461 8207 tablet_bootstrap.cc:492] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:05.386659 7284 raft_consensus.cc:2932] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:05.387660 8187 log.cc:1196] T c31148e9e4dd42c3a11787d9384c79c2 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/c31148e9e4dd42c3a11787d9384c79c2
I20240501 20:10:05.387974 8200 tablet.cc:1620] T 040734285a8347408f515daf91b43a55 P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:05.388262 8102 ts_tablet_manager.cc:1398] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 20:10:05.389247 8207 tablet_bootstrap.cc:654] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:05.390951 7974 raft_consensus.cc:482] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:05.391130 7974 raft_consensus.cc:504] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:05.391711 8202 tablet_bootstrap.cc:654] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:05.391840 7974 leader_election.cc:288] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:05.392529 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f63c61d85524d87a4b911c321f486c4" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
W20240501 20:10:05.393155 7222 leader_election.cc:341] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:05.394759 8206 tablet_bootstrap.cc:492] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:05.394817 4235 consensus_queue.cc:237] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:05.396703 8206 ts_tablet_manager.cc:1367] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.040s user 0.000s sys 0.004s
I20240501 20:10:05.376719 4627 raft_consensus.cc:2932] T dfaa3794205d482c90aa4f860fa99cf3 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:05.398100 8206 raft_consensus.cc:348] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:05.398731 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f63c61d85524d87a4b911c321f486c4" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:05.398967 8206 raft_consensus.cc:374] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:05.399111 8206 raft_consensus.cc:729] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
W20240501 20:10:05.399417 7223 leader_election.cc:341] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:05.399425 8206 consensus_queue.cc:260] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:05.399605 7223 leader_election.cc:302] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97; no voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:05.400080 7974 raft_consensus.cc:2726] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:05.405388 8215 raft_consensus.cc:1050] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530: attempting to promote NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 to VOTER
I20240501 20:10:05.406527 8098 tablet_bootstrap.cc:492] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:05.406726 8098 ts_tablet_manager.cc:1367] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.083s user 0.002s sys 0.010s
I20240501 20:10:05.407263 8206 ts_tablet_manager.cc:1398] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240501 20:10:05.407589 8098 raft_consensus.cc:348] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:05.407776 8098 raft_consensus.cc:374] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:05.407845 8098 raft_consensus.cc:729] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.408133 8098 consensus_queue.cc:260] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:05.408349 8215 consensus_queue.cc:237] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:05.409786 8098 ts_tablet_manager.cc:1398] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:10:05.411792 8102 tablet_bootstrap.cc:492] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:05.412856 8202 tablet_bootstrap.cc:492] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:05.413776 8202 ts_tablet_manager.cc:1367] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.000s
I20240501 20:10:05.414825 8202 raft_consensus.cc:348] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:05.415151 8202 raft_consensus.cc:374] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:05.415411 8202 raft_consensus.cc:729] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.415756 8202 consensus_queue.cc:260] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:05.416451 8202 ts_tablet_manager.cc:1398] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:05.419165 4498 raft_consensus.cc:1261] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:05.420171 7287 raft_consensus.cc:1261] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:05.421072 8214 consensus_queue.cc:1035] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:05.423733 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet c31148e9e4dd42c3a11787d9384c79c2 (table rebalance_test_table_15 [id=2478c1c300d14f2a9826930dc16a5dec]) successfully deleted
I20240501 20:10:05.424687 8214 consensus_queue.cc:1035] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:05.426999 8201 tablet_bootstrap.cc:492] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:05.427685 8201 ts_tablet_manager.cc:1367] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.066s user 0.009s sys 0.003s
I20240501 20:10:05.430047 7657 raft_consensus.cc:2932] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:05.430541 4498 raft_consensus.cc:2932] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:05.431957 8203 tablet_bootstrap.cc:492] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:10:05.432145 8203 ts_tablet_manager.cc:1367] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.065s user 0.004s sys 0.008s
I20240501 20:10:05.433054 8203 raft_consensus.cc:348] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:05.433336 8203 raft_consensus.cc:374] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:05.433456 8203 raft_consensus.cc:729] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.433774 8203 consensus_queue.cc:260] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:05.435973 8201 raft_consensus.cc:348] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:05.436332 8201 raft_consensus.cc:374] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:05.436488 8201 raft_consensus.cc:729] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.436843 8201 consensus_queue.cc:260] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:05.439112 8201 ts_tablet_manager.cc:1398] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 20:10:05.439246 8203 ts_tablet_manager.cc:1398] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240501 20:10:05.444331 7284 raft_consensus.cc:2932] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:05.445153 4091 catalog_manager.cc:5484] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 5 to 6, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:05.446480 8207 tablet_bootstrap.cc:492] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:05.448506 8207 ts_tablet_manager.cc:1367] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.000s
I20240501 20:10:05.450783 4092 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:05.454771 8207 raft_consensus.cc:348] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:05.455020 8207 raft_consensus.cc:374] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:05.455111 8207 raft_consensus.cc:729] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.455387 8207 consensus_queue.cc:260] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:05.456712 8207 ts_tablet_manager.cc:1398] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 20:10:05.457013 8207 tablet_bootstrap.cc:492] T 6e3d6a53365f4eceac1d99d4cd51153b P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:05.460134 8207 tablet_bootstrap.cc:654] T 6e3d6a53365f4eceac1d99d4cd51153b P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:05.461277 8200 tablet.cc:1722] T 040734285a8347408f515daf91b43a55 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:05.461917 8200 tablet.cc:1620] T 040734285a8347408f515daf91b43a55 P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:05.462301 8192 tablet.cc:1722] T 040734285a8347408f515daf91b43a55 P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:05.462863 8192 tablet.cc:1620] T 040734285a8347408f515daf91b43a55 P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:05.466210 8188 tablet.cc:1722] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:05.466749 8188 tablet.cc:1620] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:05.470510 8207 tablet_bootstrap.cc:492] T 6e3d6a53365f4eceac1d99d4cd51153b P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:05.470793 8207 ts_tablet_manager.cc:1367] T 6e3d6a53365f4eceac1d99d4cd51153b P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240501 20:10:05.471637 8207 raft_consensus.cc:348] T 6e3d6a53365f4eceac1d99d4cd51153b P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:05.471961 8207 raft_consensus.cc:374] T 6e3d6a53365f4eceac1d99d4cd51153b P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:05.472102 8207 raft_consensus.cc:729] T 6e3d6a53365f4eceac1d99d4cd51153b P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.472447 8207 consensus_queue.cc:260] T 6e3d6a53365f4eceac1d99d4cd51153b P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:05.473171 8207 ts_tablet_manager.cc:1398] T 6e3d6a53365f4eceac1d99d4cd51153b P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:05.482602 4092 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=c5969daedc4945f385966e8e481f3e7a] alter complete (version 2)
W20240501 20:10:05.492592 4169 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.67:34595, user_credentials={real_user=slave}} blocked reactor thread for 58030.7us
I20240501 20:10:05.493052 8214 raft_consensus.cc:482] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:05.493286 8214 raft_consensus.cc:504] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:05.493916 8102 tablet_bootstrap.cc:654] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:05.495201 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet dfaa3794205d482c90aa4f860fa99cf3 with delete_type TABLET_DATA_TOMBSTONED (TS 1d34e18a1d394d3f91c9f50d1c7bea4e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:05.495398 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ddbf0c6a4e4c1fb3c8d59288894a35" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:05.495676 7420 raft_consensus.cc:2454] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:05.496224 4170 leader_election.cc:302] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:05.496765 8214 leader_election.cc:288] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:05.497273 8219 raft_consensus.cc:2781] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:05.497426 8219 raft_consensus.cc:482] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:05.497556 8219 raft_consensus.cc:3037] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:05.503341 8219 raft_consensus.cc:504] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:05.504740 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ddbf0c6a4e4c1fb3c8d59288894a35" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:05.505069 7420 raft_consensus.cc:3037] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:05.507994 8221 tablet_replica.cc:330] T dfaa3794205d482c90aa4f860fa99cf3 P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:05.508332 8221 raft_consensus.cc:2229] T dfaa3794205d482c90aa4f860fa99cf3 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:05.508644 8221 raft_consensus.cc:2258] T dfaa3794205d482c90aa4f860fa99cf3 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:05.509054 7420 raft_consensus.cc:2454] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:05.509743 4170 leader_election.cc:302] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:05.509867 8221 ts_tablet_manager.cc:1875] T dfaa3794205d482c90aa4f860fa99cf3 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:05.523732 8224 raft_consensus.cc:2781] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:05.524026 8224 raft_consensus.cc:686] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:10:05.524477 8224 consensus_queue.cc:237] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:05.527338 7154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ddbf0c6a4e4c1fb3c8d59288894a35" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:05.527612 7154 raft_consensus.cc:2454] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:05.532737 8221 ts_tablet_manager.cc:1888] T dfaa3794205d482c90aa4f860fa99cf3 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:05.532961 8221 log.cc:1196] T dfaa3794205d482c90aa4f860fa99cf3 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/dfaa3794205d482c90aa4f860fa99cf3
I20240501 20:10:05.533879 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet dfaa3794205d482c90aa4f860fa99cf3 (table rebalance_test_table_15 [id=2478c1c300d14f2a9826930dc16a5dec]) successfully deleted
I20240501 20:10:05.533982 7154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ddbf0c6a4e4c1fb3c8d59288894a35" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:05.534284 7154 raft_consensus.cc:3037] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:05.538405 7154 raft_consensus.cc:2454] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:05.544286 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dfaa3794205d482c90aa4f860fa99cf3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:05.545174 8219 leader_election.cc:288] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:05.548404 8102 tablet_bootstrap.cc:492] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:05.548725 8102 ts_tablet_manager.cc:1367] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.137s user 0.003s sys 0.003s
I20240501 20:10:05.551959 8102 raft_consensus.cc:348] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:05.552721 8102 raft_consensus.cc:374] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:05.552922 8102 raft_consensus.cc:729] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.553304 8102 consensus_queue.cc:260] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:05.555868 8102 ts_tablet_manager.cc:1398] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240501 20:10:05.560644 7420 raft_consensus.cc:1261] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:05.561461 4365 raft_consensus.cc:1261] T 34cb2f3c90cc4158a61f426c0e5cb86b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:05.562383 4498 raft_consensus.cc:1261] T 34cb2f3c90cc4158a61f426c0e5cb86b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:05.561858 8225 consensus_queue.cc:1035] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 20:10:05.562662 8219 raft_consensus.cc:1050] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530: attempting to promote NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d to VOTER
I20240501 20:10:05.563362 4091 catalog_manager.cc:5484] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:05.563481 8219 consensus_queue.cc:237] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:05.564692 8211 consensus_queue.cc:1035] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:05.565492 8211 consensus_queue.cc:1035] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:05.580322 7022 raft_consensus.cc:1261] T 9d634c1feb7449f686cf3af2a07181e0 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:05.581583 8219 consensus_queue.cc:1035] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:05.583109 4367 raft_consensus.cc:1261] T 9d634c1feb7449f686cf3af2a07181e0 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:05.586243 4087 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:05.589498 8215 raft_consensus.cc:2932] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:05.596076 7420 raft_consensus.cc:2932] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:05.596910 7974 raft_consensus.cc:482] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:05.597105 7974 raft_consensus.cc:504] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:05.597826 7974 leader_election.cc:288] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:05.599311 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3d6a53365f4eceac1d99d4cd51153b" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:05.599546 7155 raft_consensus.cc:2454] T 6e3d6a53365f4eceac1d99d4cd51153b P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:05.599929 8175 raft_consensus.cc:482] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:05.600082 8175 raft_consensus.cc:504] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:05.600867 8175 leader_election.cc:288] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:05.603641 8211 consensus_queue.cc:1035] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:05.603886 8188 tablet.cc:1722] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:05.606382 7220 leader_election.cc:302] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:05.606892 8210 raft_consensus.cc:2781] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:05.606984 8210 raft_consensus.cc:482] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:05.607075 8210 raft_consensus.cc:3037] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:05.609982 8210 raft_consensus.cc:504] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:05.611774 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3d6a53365f4eceac1d99d4cd51153b" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:05.612028 7155 raft_consensus.cc:2454] T 6e3d6a53365f4eceac1d99d4cd51153b P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:10:05.613090 4091 catalog_manager.cc:5484] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:05.615536 4497 raft_consensus.cc:2932] T 34cb2f3c90cc4158a61f426c0e5cb86b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:05.619251 4299 leader_election.cc:302] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:05.620652 8192 tablet.cc:1722] T 040734285a8347408f515daf91b43a55 P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:05.623704 4498 raft_consensus.cc:1261] T 9d634c1feb7449f686cf3af2a07181e0 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:05.624210 8175 raft_consensus.cc:2781] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:05.624332 8175 raft_consensus.cc:482] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:05.624428 8175 raft_consensus.cc:3037] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:05.627506 4235 consensus_queue.cc:237] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:05.630700 8175 raft_consensus.cc:504] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:05.631381 8175 leader_election.cc:288] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Requested vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:05.633978 4368 raft_consensus.cc:2932] T 34cb2f3c90cc4158a61f426c0e5cb86b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:05.634481 8224 consensus_queue.cc:1035] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:05.638837 4367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3d6a53365f4eceac1d99d4cd51153b" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:10:05.639129 4367 raft_consensus.cc:2379] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bceeae1c79964acabdae8ff771bf0c97 in current term 1: Already voted for candidate 10b47782558c41ee91f5e164a372049a in this term.
I20240501 20:10:05.643412 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3d6a53365f4eceac1d99d4cd51153b" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:05.643685 7153 raft_consensus.cc:3037] T 6e3d6a53365f4eceac1d99d4cd51153b P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:05.643996 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3d6a53365f4eceac1d99d4cd51153b" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:05.645184 7421 raft_consensus.cc:1261] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:05.645918 8200 tablet.cc:1722] T 040734285a8347408f515daf91b43a55 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:05.646509 7153 raft_consensus.cc:2454] T 6e3d6a53365f4eceac1d99d4cd51153b P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:10:05.650849 4299 leader_election.cc:302] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:05.651160 8133 raft_consensus.cc:2781] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:05.651321 8133 raft_consensus.cc:686] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Becoming Leader. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Running, Role: LEADER
I20240501 20:10:05.651587 8133 consensus_queue.cc:237] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:05.654071 4367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3d6a53365f4eceac1d99d4cd51153b" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:10:05.655066 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3d6a53365f4eceac1d99d4cd51153b" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:05.655380 7287 raft_consensus.cc:2379] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 10b47782558c41ee91f5e164a372049a in current term 1: Already voted for candidate bceeae1c79964acabdae8ff771bf0c97 in this term.
I20240501 20:10:05.655335 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3d6a53365f4eceac1d99d4cd51153b" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:05.655579 7284 raft_consensus.cc:2379] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10b47782558c41ee91f5e164a372049a in current term 1: Already voted for candidate bceeae1c79964acabdae8ff771bf0c97 in this term.
I20240501 20:10:05.657189 7222 leader_election.cc:302] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97; no voters: 10b47782558c41ee91f5e164a372049a, db0742ee3860468590eefef6ae6074fd
I20240501 20:10:05.658134 8219 consensus_queue.cc:1035] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:05.659152 7974 raft_consensus.cc:2726] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 20:10:05.665234 8210 leader_election.cc:288] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:05.665562 4364 raft_consensus.cc:1261] T 34cb2f3c90cc4158a61f426c0e5cb86b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:05.666106 8215 raft_consensus.cc:2932] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:05.667037 8225 raft_consensus.cc:2932] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:05.669478 4498 consensus_queue.cc:237] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:05.671686 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 34cb2f3c90cc4158a61f426c0e5cb86b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:05.673123 7420 raft_consensus.cc:2932] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:05.678351 8225 consensus_queue.cc:1035] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:05.683029 4092 catalog_manager.cc:5484] T 34cb2f3c90cc4158a61f426c0e5cb86b P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:05.679955 4091 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table_new_name [id=c5969daedc4945f385966e8e481f3e7a] alter complete (version 3)
I20240501 20:10:05.690912 7022 raft_consensus.cc:2932] T 9d634c1feb7449f686cf3af2a07181e0 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:05.692281 4364 raft_consensus.cc:2932] T 9d634c1feb7449f686cf3af2a07181e0 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:05.696754 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet 34cb2f3c90cc4158a61f426c0e5cb86b with delete_type TABLET_DATA_TOMBSTONED (TS 3e974c03d3c546f9ab6c91e762a5426b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:05.697129 8187 tablet_replica.cc:330] T 34cb2f3c90cc4158a61f426c0e5cb86b P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:05.697322 8187 raft_consensus.cc:2229] T 34cb2f3c90cc4158a61f426c0e5cb86b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:05.697551 8187 raft_consensus.cc:2258] T 34cb2f3c90cc4158a61f426c0e5cb86b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:05.698856 4364 raft_consensus.cc:2932] T 34cb2f3c90cc4158a61f426c0e5cb86b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:05.699295 8187 ts_tablet_manager.cc:1875] T 34cb2f3c90cc4158a61f426c0e5cb86b P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:05.703428 4497 raft_consensus.cc:2932] T 9d634c1feb7449f686cf3af2a07181e0 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:05.698484 4087 catalog_manager.cc:5484] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a reported cstate change: term changed from 0 to 1, leader changed from <none> to 10b47782558c41ee91f5e164a372049a (127.29.219.66). New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:05.709262 4083 catalog_manager.cc:5484] T 9d634c1feb7449f686cf3af2a07181e0 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:05.732151 8187 ts_tablet_manager.cc:1888] T 34cb2f3c90cc4158a61f426c0e5cb86b P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:05.733822 8187 log.cc:1196] T 34cb2f3c90cc4158a61f426c0e5cb86b P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/34cb2f3c90cc4158a61f426c0e5cb86b
I20240501 20:10:05.735941 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet 34cb2f3c90cc4158a61f426c0e5cb86b (table rebalance_test_table_11 [id=3b02a4c6c0f54681a87f5ca2381de01d]) successfully deleted
W20240501 20:10:05.741977 4432 consensus_peers.cc:485] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b -> Peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Couldn't send request to peer db0742ee3860468590eefef6ae6074fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d165c29dabe4ae3be11ed84702c3cb2. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:05.742336 4234 consensus_queue.cc:237] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:05.743721 4364 raft_consensus.cc:1261] T 9d165c29dabe4ae3be11ed84702c3cb2 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:05.745209 4627 raft_consensus.cc:1261] T 9d165c29dabe4ae3be11ed84702c3cb2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:05.745834 7919 consensus_queue.cc:1035] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:05.746183 8153 consensus_queue.cc:1035] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:05.746470 7022 raft_consensus.cc:1261] T 9d634c1feb7449f686cf3af2a07181e0 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:05.747300 7880 raft_consensus.cc:482] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:05.747464 7880 raft_consensus.cc:504] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:05.748093 7880 leader_election.cc:288] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:05.748476 8215 consensus_queue.cc:1035] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:05.748735 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f63c61d85524d87a4b911c321f486c4" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:05.748956 7284 raft_consensus.cc:2454] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:05.749490 6956 leader_election.cc:302] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:05.749836 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f63c61d85524d87a4b911c321f486c4" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:05.750023 4630 raft_consensus.cc:2454] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:05.752861 4498 raft_consensus.cc:1261] T 9d634c1feb7449f686cf3af2a07181e0 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:05.753520 8215 consensus_queue.cc:1035] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:05.754204 7880 raft_consensus.cc:2781] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:05.754360 7880 raft_consensus.cc:482] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:05.756623 7880 raft_consensus.cc:3037] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:05.757843 8215 raft_consensus.cc:2932] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:05.759676 7880 raft_consensus.cc:504] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:05.761548 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f63c61d85524d87a4b911c321f486c4" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:10:05.761848 4630 raft_consensus.cc:3037] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:05.762895 4497 raft_consensus.cc:2932] T 9d634c1feb7449f686cf3af2a07181e0 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:05.765314 4630 raft_consensus.cc:2454] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:05.766095 6958 leader_election.cc:302] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:05.766638 8163 raft_consensus.cc:2781] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:05.766803 8163 raft_consensus.cc:686] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Becoming Leader. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Running, Role: LEADER
I20240501 20:10:05.767241 8163 consensus_queue.cc:237] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:05.768641 8153 raft_consensus.cc:2932] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:05.769331 4364 raft_consensus.cc:2932] T 9d165c29dabe4ae3be11ed84702c3cb2 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:05.771188 7022 raft_consensus.cc:2932] T 9d634c1feb7449f686cf3af2a07181e0 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:05.774956 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f63c61d85524d87a4b911c321f486c4" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:05.775206 7284 raft_consensus.cc:3037] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:05.776710 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9d634c1feb7449f686cf3af2a07181e0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:05.776787 4084 catalog_manager.cc:5484] T 9d165c29dabe4ae3be11ed84702c3cb2 P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:05.778299 7880 leader_election.cc:288] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:05.790524 4092 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:05.790835 4092 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:05.793238 4091 catalog_manager.cc:5484] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:05.794325 4083 catalog_manager.cc:5484] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69). New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:05.802470 4092 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 040734285a8347408f515daf91b43a55
I20240501 20:10:05.804279 7401 tablet_service.cc:1514] Processing DeleteTablet for tablet 040734285a8347408f515daf91b43a55 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:05 UTC) from {username='slave'} at 127.0.0.1:40182
I20240501 20:10:05.805261 7000 tablet_service.cc:1514] Processing DeleteTablet for tablet 040734285a8347408f515daf91b43a55 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:05 UTC) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:05.807956 7266 tablet_service.cc:1514] Processing DeleteTablet for tablet 040734285a8347408f515daf91b43a55 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:05 UTC) from {username='slave'} at 127.0.0.1:54166
I20240501 20:10:05.814556 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d634c1feb7449f686cf3af2a07181e0 with delete_type TABLET_DATA_TOMBSTONED (TS 10b47782558c41ee91f5e164a372049a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:05.824123 7284 raft_consensus.cc:2454] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:05.830121 8256 tablet_replica.cc:330] T 040734285a8347408f515daf91b43a55 P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:05.830606 8256 raft_consensus.cc:2229] T 040734285a8347408f515daf91b43a55 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:05.830950 8256 raft_consensus.cc:2258] T 040734285a8347408f515daf91b43a55 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:05.831964 8256 ts_tablet_manager.cc:1875] T 040734285a8347408f515daf91b43a55 P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:05.839557 8256 ts_tablet_manager.cc:1888] T 040734285a8347408f515daf91b43a55 P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:05.846211 8256 log.cc:1196] T 040734285a8347408f515daf91b43a55 P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/040734285a8347408f515daf91b43a55
I20240501 20:10:05.847105 8256 ts_tablet_manager.cc:1909] T 040734285a8347408f515daf91b43a55 P bceeae1c79964acabdae8ff771bf0c97: Deleting consensus metadata
I20240501 20:10:05.849030 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet 040734285a8347408f515daf91b43a55 (table rebalancer_dynamic_table_new_name [id=c5969daedc4945f385966e8e481f3e7a]) successfully deleted
I20240501 20:10:05.872548 4627 raft_consensus.cc:2932] T 9d165c29dabe4ae3be11ed84702c3cb2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:05.883314 8257 tablet_replica.cc:330] T 9d634c1feb7449f686cf3af2a07181e0 P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:05.883574 8257 raft_consensus.cc:2229] T 9d634c1feb7449f686cf3af2a07181e0 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:05.884037 8257 raft_consensus.cc:2258] T 9d634c1feb7449f686cf3af2a07181e0 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:05.885461 8257 ts_tablet_manager.cc:1875] T 9d634c1feb7449f686cf3af2a07181e0 P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:05.891216 8254 tablet_replica.cc:330] T 040734285a8347408f515daf91b43a55 P b9957d4086bc4a76b18ed9265a5739d5: stopping tablet replica
I20240501 20:10:05.891585 8254 raft_consensus.cc:2229] T 040734285a8347408f515daf91b43a55 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:05.891952 8254 raft_consensus.cc:2258] T 040734285a8347408f515daf91b43a55 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:05.892628 8255 tablet_replica.cc:330] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d: stopping tablet replica
I20240501 20:10:05.892943 8255 raft_consensus.cc:2229] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:05.893508 8255 raft_consensus.cc:2258] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:05.893685 8254 ts_tablet_manager.cc:1875] T 040734285a8347408f515daf91b43a55 P b9957d4086bc4a76b18ed9265a5739d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:05.899286 8255 ts_tablet_manager.cc:1875] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:05.902009 8257 ts_tablet_manager.cc:1888] T 9d634c1feb7449f686cf3af2a07181e0 P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:05.902236 8257 log.cc:1196] T 9d634c1feb7449f686cf3af2a07181e0 P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/9d634c1feb7449f686cf3af2a07181e0
I20240501 20:10:05.903983 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet 9d634c1feb7449f686cf3af2a07181e0 (table rebalance_test_table_5 [id=1e137c7380c64655a3b011211c536d72]) successfully deleted
I20240501 20:10:05.911674 4091 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:05.916491 8254 ts_tablet_manager.cc:1888] T 040734285a8347408f515daf91b43a55 P b9957d4086bc4a76b18ed9265a5739d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:05.916873 8254 log.cc:1196] T 040734285a8347408f515daf91b43a55 P b9957d4086bc4a76b18ed9265a5739d5: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/wals/040734285a8347408f515daf91b43a55
I20240501 20:10:05.917491 8254 ts_tablet_manager.cc:1909] T 040734285a8347408f515daf91b43a55 P b9957d4086bc4a76b18ed9265a5739d5: Deleting consensus metadata
I20240501 20:10:05.918138 8255 ts_tablet_manager.cc:1888] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:05.918390 8255 log.cc:1196] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/wals/040734285a8347408f515daf91b43a55
I20240501 20:10:05.918830 4079 catalog_manager.cc:4830] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): tablet 040734285a8347408f515daf91b43a55 (table rebalancer_dynamic_table_new_name [id=c5969daedc4945f385966e8e481f3e7a]) successfully deleted
I20240501 20:10:05.919008 8255 ts_tablet_manager.cc:1909] T 040734285a8347408f515daf91b43a55 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting consensus metadata
I20240501 20:10:05.921252 4478 tablet_service.cc:1467] Processing CreateTablet for tablet 2c1f95b56c56409990c7aee0f8a9fb8f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7fff78ff161d4f66bcc8ca9dc1940a47]), partition=
I20240501 20:10:05.921692 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1f95b56c56409990c7aee0f8a9fb8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.922559 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 2c1f95b56c56409990c7aee0f8a9fb8f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7fff78ff161d4f66bcc8ca9dc1940a47]), partition=
I20240501 20:10:05.922943 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1f95b56c56409990c7aee0f8a9fb8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.923028 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet 040734285a8347408f515daf91b43a55 (table rebalancer_dynamic_table_new_name [id=c5969daedc4945f385966e8e481f3e7a]) successfully deleted
I20240501 20:10:05.923758 7000 tablet_service.cc:1467] Processing CreateTablet for tablet 2c1f95b56c56409990c7aee0f8a9fb8f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7fff78ff161d4f66bcc8ca9dc1940a47]), partition=
I20240501 20:10:05.924106 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c1f95b56c56409990c7aee0f8a9fb8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:05.931615 8264 tablet_bootstrap.cc:492] T 2c1f95b56c56409990c7aee0f8a9fb8f P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:05.933498 8264 tablet_bootstrap.cc:654] T 2c1f95b56c56409990c7aee0f8a9fb8f P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:05.936287 4364 consensus_queue.cc:237] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:05.939364 8265 tablet_bootstrap.cc:492] T 2c1f95b56c56409990c7aee0f8a9fb8f P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:05.940359 4234 raft_consensus.cc:1261] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:05.940940 8175 consensus_queue.cc:1035] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:05.942941 4497 raft_consensus.cc:1261] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:05.945636 4300 consensus_peers.cc:485] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a -> Peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Couldn't send request to peer bceeae1c79964acabdae8ff771bf0c97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f40b062e103d4e54a913fa0f4111f623. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:05.945698 8265 tablet_bootstrap.cc:654] T 2c1f95b56c56409990c7aee0f8a9fb8f P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:05.947767 7335 consensus_queue.cc:1035] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:05.947939 8263 tablet_bootstrap.cc:492] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:05.950544 8263 tablet_bootstrap.cc:654] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:05.951198 8175 raft_consensus.cc:2932] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:05.955294 4497 raft_consensus.cc:2932] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:05.958580 8264 tablet_bootstrap.cc:492] T 2c1f95b56c56409990c7aee0f8a9fb8f P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:05.959483 8264 ts_tablet_manager.cc:1367] T 2c1f95b56c56409990c7aee0f8a9fb8f P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.005s
I20240501 20:10:05.960345 4084 catalog_manager.cc:5484] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:05.960502 8264 raft_consensus.cc:348] T 2c1f95b56c56409990c7aee0f8a9fb8f P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:05.960762 8264 raft_consensus.cc:374] T 2c1f95b56c56409990c7aee0f8a9fb8f P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:05.960868 8264 raft_consensus.cc:729] T 2c1f95b56c56409990c7aee0f8a9fb8f P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.964020 8264 consensus_queue.cc:260] T 2c1f95b56c56409990c7aee0f8a9fb8f P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:05.964574 8263 tablet_bootstrap.cc:492] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:10:05.964780 8263 ts_tablet_manager.cc:1367] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20240501 20:10:05.965628 8263 raft_consensus.cc:348] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:05.965806 8264 ts_tablet_manager.cc:1398] T 2c1f95b56c56409990c7aee0f8a9fb8f P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 20:10:05.966081 8263 raft_consensus.cc:374] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:05.966213 8263 raft_consensus.cc:729] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.966596 8263 consensus_queue.cc:260] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:05.966923 8265 tablet_bootstrap.cc:492] T 2c1f95b56c56409990c7aee0f8a9fb8f P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:05.967095 8265 ts_tablet_manager.cc:1367] T 2c1f95b56c56409990c7aee0f8a9fb8f P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.009s
I20240501 20:10:05.968077 8263 ts_tablet_manager.cc:1398] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:10:05.968535 8265 raft_consensus.cc:348] T 2c1f95b56c56409990c7aee0f8a9fb8f P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:05.968765 8265 raft_consensus.cc:374] T 2c1f95b56c56409990c7aee0f8a9fb8f P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:05.968894 8265 raft_consensus.cc:729] T 2c1f95b56c56409990c7aee0f8a9fb8f P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:05.969174 8265 consensus_queue.cc:260] T 2c1f95b56c56409990c7aee0f8a9fb8f P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:05.969029 4234 raft_consensus.cc:2932] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:05.969873 8265 ts_tablet_manager.cc:1398] T 2c1f95b56c56409990c7aee0f8a9fb8f P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:10:05.989125 4498 consensus_queue.cc:237] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:05.990759 8215 consensus_queue.cc:1035] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:05.992074 4364 raft_consensus.cc:1261] T 644d4d293e8c462297c6419965bd2c89 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:05.992112 4234 raft_consensus.cc:1261] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:05.992772 8250 consensus_queue.cc:1035] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:05.993144 8251 consensus_queue.cc:1035] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:10:05.994088 4434 consensus_peers.cc:485] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 644d4d293e8c462297c6419965bd2c89. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:06.008864 8250 raft_consensus.cc:2932] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:06.014142 8215 consensus_queue.cc:1035] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:06.013973 4365 raft_consensus.cc:2932] T 644d4d293e8c462297c6419965bd2c89 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:06.014585 4234 raft_consensus.cc:2932] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:06.021374 4091 catalog_manager.cc:5484] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
W20240501 20:10:06.030128 4545 tablet.cc:2369] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:10:06.043828 4498 consensus_queue.cc:237] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:06.046204 4364 raft_consensus.cc:1261] T ef7c0a48f11945d3953f3461d06bc184 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:06.046835 4627 raft_consensus.cc:1261] T ef7c0a48f11945d3953f3461d06bc184 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:06.046989 8153 consensus_queue.cc:1035] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:06.047384 8251 consensus_queue.cc:1035] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:06.050398 8153 raft_consensus.cc:2932] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:06.051376 4364 raft_consensus.cc:2932] T ef7c0a48f11945d3953f3461d06bc184 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:06.051550 4627 raft_consensus.cc:2932] T ef7c0a48f11945d3953f3461d06bc184 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:06.057500 4087 catalog_manager.cc:5484] T ef7c0a48f11945d3953f3461d06bc184 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
W20240501 20:10:06.060791 4435 consensus_peers.cc:485] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b -> Peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): Couldn't send request to peer b9957d4086bc4a76b18ed9265a5739d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef7c0a48f11945d3953f3461d06bc184. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:06.065418 7335 consensus_queue.cc:1035] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:06.077253 8251 raft_consensus.cc:482] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:06.077527 8251 raft_consensus.cc:504] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:06.078738 8251 leader_election.cc:288] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:06.079109 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1f95b56c56409990c7aee0f8a9fb8f" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:06.079384 7420 raft_consensus.cc:2454] T 2c1f95b56c56409990c7aee0f8a9fb8f P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:10:06.079924 4435 leader_election.cc:302] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, b9957d4086bc4a76b18ed9265a5739d5; no voters:
I20240501 20:10:06.079957 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1f95b56c56409990c7aee0f8a9fb8f" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:06.080173 7022 raft_consensus.cc:2454] T 2c1f95b56c56409990c7aee0f8a9fb8f P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:10:06.080858 8251 raft_consensus.cc:2781] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:06.081028 8251 raft_consensus.cc:482] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:06.081154 8251 raft_consensus.cc:3037] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:06.084525 8251 raft_consensus.cc:504] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:06.084622 7335 consensus_queue.cc:1035] T 6e3d6a53365f4eceac1d99d4cd51153b P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:06.087679 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1f95b56c56409990c7aee0f8a9fb8f" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:06.087689 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1f95b56c56409990c7aee0f8a9fb8f" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:06.087939 7022 raft_consensus.cc:3037] T 2c1f95b56c56409990c7aee0f8a9fb8f P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:06.087951 7420 raft_consensus.cc:3037] T 2c1f95b56c56409990c7aee0f8a9fb8f P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:06.091092 7420 raft_consensus.cc:2454] T 2c1f95b56c56409990c7aee0f8a9fb8f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:10:06.091111 7022 raft_consensus.cc:2454] T 2c1f95b56c56409990c7aee0f8a9fb8f P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:10:06.091723 4435 leader_election.cc:302] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, b9957d4086bc4a76b18ed9265a5739d5; no voters:
I20240501 20:10:06.092105 8153 raft_consensus.cc:2781] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:06.092239 8153 raft_consensus.cc:686] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:10:06.092739 8153 consensus_queue.cc:237] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:06.095566 8251 leader_election.cc:288] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Requested vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:06.099699 4365 consensus_queue.cc:237] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:06.109588 4498 raft_consensus.cc:1261] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:06.110546 4300 consensus_peers.cc:485] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a -> Peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Couldn't send request to peer bceeae1c79964acabdae8ff771bf0c97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c66514deec5744da832bc030a582c46a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:06.111538 7335 consensus_queue.cc:1035] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 20:10:06.115108 4627 raft_consensus.cc:1261] T c66514deec5744da832bc030a582c46a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:06.115820 7335 consensus_queue.cc:1035] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:06.115496 8175 raft_consensus.cc:2932] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:06.116554 4497 raft_consensus.cc:2932] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:06.121392 4084 catalog_manager.cc:5484] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:06.128280 4087 catalog_manager.cc:5484] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:06.142508 4627 raft_consensus.cc:2932] T c66514deec5744da832bc030a582c46a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:06.163455 4627 consensus_queue.cc:237] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:06.166991 4497 raft_consensus.cc:1261] T 7725fa9f98e9439797a532ed6f2f7dba P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:06.167483 4234 raft_consensus.cc:1261] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:06.167686 8071 consensus_queue.cc:1035] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:10:06.168120 4562 consensus_peers.cc:485] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Couldn't send request to peer db0742ee3860468590eefef6ae6074fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7725fa9f98e9439797a532ed6f2f7dba. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:06.168339 8071 consensus_queue.cc:1035] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:06.171609 8314 raft_consensus.cc:2932] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:06.172681 4497 raft_consensus.cc:2932] T 7725fa9f98e9439797a532ed6f2f7dba P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:06.175937 4234 raft_consensus.cc:2932] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:06.182036 4084 catalog_manager.cc:5484] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:06.205894 4627 consensus_queue.cc:237] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:06.208328 4234 raft_consensus.cc:1261] T fcff4be357084bf0903cfb1c0f62a866 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:06.208653 4564 consensus_peers.cc:485] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fcff4be357084bf0903cfb1c0f62a866. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:06.209194 8314 consensus_queue.cc:1035] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:06.209411 4497 raft_consensus.cc:1261] T fcff4be357084bf0903cfb1c0f62a866 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:06.210191 8314 consensus_queue.cc:1035] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:06.212949 8314 raft_consensus.cc:2932] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:06.215293 4234 raft_consensus.cc:2932] T fcff4be357084bf0903cfb1c0f62a866 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:06.215981 4497 raft_consensus.cc:2932] T fcff4be357084bf0903cfb1c0f62a866 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:06.221207 4084 catalog_manager.cc:5484] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:06.243583 4234 consensus_queue.cc:237] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:06.245870 4365 raft_consensus.cc:1261] T d690199665174c7d9bdbd54d2af4ac70 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:06.245875 4497 raft_consensus.cc:1261] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:06.246330 4170 consensus_peers.cc:485] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 -> Peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): Couldn't send request to peer b9957d4086bc4a76b18ed9265a5739d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d690199665174c7d9bdbd54d2af4ac70. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:06.246423 8225 consensus_queue.cc:1035] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:06.246778 8219 consensus_queue.cc:1035] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:06.249589 8219 raft_consensus.cc:2932] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:06.250278 4365 raft_consensus.cc:2932] T d690199665174c7d9bdbd54d2af4ac70 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:06.250969 4497 raft_consensus.cc:2932] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:06.256338 4087 catalog_manager.cc:5484] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:06.259485 4092 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:06.267496 7420 raft_consensus.cc:1261] T 2c1f95b56c56409990c7aee0f8a9fb8f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:06.268191 8251 consensus_queue.cc:1035] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:06.271293 7022 raft_consensus.cc:1261] T 2c1f95b56c56409990c7aee0f8a9fb8f P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:06.273317 8251 consensus_queue.cc:1035] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:06.295437 4092 catalog_manager.cc:5576] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817) does not have the latest schema for tablet 2c1f95b56c56409990c7aee0f8a9fb8f (table rebalancer_dynamic_table [id=7fff78ff161d4f66bcc8ca9dc1940a47]). Expected version 1 got 0
I20240501 20:10:06.301055 8346 tablet.cc:1722] T 2c1f95b56c56409990c7aee0f8a9fb8f P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:06.302634 8346 tablet.cc:1620] T 2c1f95b56c56409990c7aee0f8a9fb8f P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:06.304291 4092 catalog_manager.cc:5576] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817) does not have the latest schema for tablet 2c1f95b56c56409990c7aee0f8a9fb8f (table rebalancer_dynamic_table [id=7fff78ff161d4f66bcc8ca9dc1940a47]). Expected version 1 got 0
I20240501 20:10:06.311781 8345 tablet.cc:1722] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:06.312258 8163 consensus_queue.cc:1035] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:06.316666 8345 tablet.cc:1620] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b: MemRowSet was empty: no flush needed.
I20240501 20:10:06.322778 4091 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=7fff78ff161d4f66bcc8ca9dc1940a47] alter complete (version 1)
I20240501 20:10:06.329983 8163 consensus_queue.cc:1035] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:06.342021 4092 catalog_manager.cc:5576] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237) does not have the latest schema for tablet 2c1f95b56c56409990c7aee0f8a9fb8f (table rebalancer_dynamic_table [id=7fff78ff161d4f66bcc8ca9dc1940a47]). Expected version 1 got 0
I20240501 20:10:06.352543 7153 consensus_queue.cc:237] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
W20240501 20:10:06.357151 7091 consensus_peers.cc:485] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd -> Peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Couldn't send request to peer 1d34e18a1d394d3f91c9f50d1c7bea4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb85d7825c564339bd554deeded45cec. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:06.358040 7284 raft_consensus.cc:1261] T bb85d7825c564339bd554deeded45cec P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:06.358048 8348 tablet.cc:1722] T 2c1f95b56c56409990c7aee0f8a9fb8f P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:06.358906 8139 consensus_queue.cc:1035] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:06.359894 7022 raft_consensus.cc:1261] T bb85d7825c564339bd554deeded45cec P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:06.360428 8139 consensus_queue.cc:1035] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:06.363626 4092 catalog_manager.cc:5576] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237) does not have the latest schema for tablet 2c1f95b56c56409990c7aee0f8a9fb8f (table rebalancer_dynamic_table [id=7fff78ff161d4f66bcc8ca9dc1940a47]). Expected version 1 got 0
I20240501 20:10:06.365125 8348 tablet.cc:1620] T 2c1f95b56c56409990c7aee0f8a9fb8f P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:06.368064 8355 ts_tablet_manager.cc:902] T 9d165c29dabe4ae3be11ed84702c3cb2 P db0742ee3860468590eefef6ae6074fd: Initiating tablet copy from peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:06.369704 8139 raft_consensus.cc:2932] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:06.372810 8349 ts_tablet_manager.cc:902] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97: Initiating tablet copy from peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:06.380513 7287 raft_consensus.cc:2932] T bb85d7825c564339bd554deeded45cec P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:06.381992 4092 catalog_manager.cc:5484] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:06.427693 4091 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:06.430742 7022 raft_consensus.cc:2932] T bb85d7825c564339bd554deeded45cec P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:06.440876 8345 tablet.cc:1722] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:06.441465 8345 tablet.cc:1620] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b: MemRowSet was empty: no flush needed.
I20240501 20:10:06.442293 8348 tablet.cc:1722] T 2c1f95b56c56409990c7aee0f8a9fb8f P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:06.442986 8348 tablet.cc:1620] T 2c1f95b56c56409990c7aee0f8a9fb8f P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:06.443312 8346 tablet.cc:1722] T 2c1f95b56c56409990c7aee0f8a9fb8f P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:06.443877 8346 tablet.cc:1620] T 2c1f95b56c56409990c7aee0f8a9fb8f P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:06.456571 4087 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=7fff78ff161d4f66bcc8ca9dc1940a47] alter complete (version 2)
I20240501 20:10:06.467273 8349 tablet_copy_client.cc:308] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.66:40689
I20240501 20:10:06.468089 4388 tablet_copy_service.cc:140] P 10b47782558c41ee91f5e164a372049a: Received BeginTabletCopySession request for tablet f40b062e103d4e54a913fa0f4111f623 from peer bceeae1c79964acabdae8ff771bf0c97 ({username='slave'} at 127.29.219.71:41437)
I20240501 20:10:06.468292 4388 tablet_copy_service.cc:161] P 10b47782558c41ee91f5e164a372049a: Beginning new tablet copy session on tablet f40b062e103d4e54a913fa0f4111f623 from peer bceeae1c79964acabdae8ff771bf0c97 at {username='slave'} at 127.29.219.71:41437: session id = bceeae1c79964acabdae8ff771bf0c97-f40b062e103d4e54a913fa0f4111f623
I20240501 20:10:06.471297 4388 tablet_copy_source_session.cc:205] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:06.473500 8355 tablet_copy_client.cc:308] T 9d165c29dabe4ae3be11ed84702c3cb2 P db0742ee3860468590eefef6ae6074fd: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.67:34595
I20240501 20:10:06.473675 8349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f40b062e103d4e54a913fa0f4111f623. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:06.474403 4519 tablet_copy_service.cc:140] P 3e974c03d3c546f9ab6c91e762a5426b: Received BeginTabletCopySession request for tablet 9d165c29dabe4ae3be11ed84702c3cb2 from peer db0742ee3860468590eefef6ae6074fd ({username='slave'} at 127.29.219.70:56941)
I20240501 20:10:06.474573 4519 tablet_copy_service.cc:161] P 3e974c03d3c546f9ab6c91e762a5426b: Beginning new tablet copy session on tablet 9d165c29dabe4ae3be11ed84702c3cb2 from peer db0742ee3860468590eefef6ae6074fd at {username='slave'} at 127.29.219.70:56941: session id = db0742ee3860468590eefef6ae6074fd-9d165c29dabe4ae3be11ed84702c3cb2
I20240501 20:10:06.477456 4519 tablet_copy_source_session.cc:205] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:06.479395 8355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d165c29dabe4ae3be11ed84702c3cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:06.482010 8349 tablet_copy_client.cc:783] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:06.482374 8349 tablet_copy_client.cc:647] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:06.484315 8349 tablet_copy_client.cc:523] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:06.486140 4627 consensus_queue.cc:237] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:06.486605 8355 tablet_copy_client.cc:783] T 9d165c29dabe4ae3be11ed84702c3cb2 P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:06.486900 8355 tablet_copy_client.cc:647] T 9d165c29dabe4ae3be11ed84702c3cb2 P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:06.488516 8349 tablet_bootstrap.cc:492] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:06.490058 8355 tablet_copy_client.cc:523] T 9d165c29dabe4ae3be11ed84702c3cb2 P db0742ee3860468590eefef6ae6074fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:06.493407 4497 raft_consensus.cc:1261] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:06.495226 4365 raft_consensus.cc:1261] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:06.495225 4562 consensus_peers.cc:485] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Couldn't send request to peer db0742ee3860468590eefef6ae6074fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41d6e6dce35e4d0da594f147a7f9bfc4. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:06.495784 8314 consensus_queue.cc:1035] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:06.496351 8314 consensus_queue.cc:1035] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:06.499207 4091 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:06.499648 8355 tablet_bootstrap.cc:492] T 9d165c29dabe4ae3be11ed84702c3cb2 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:06.513779 8374 raft_consensus.cc:2932] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:06.522333 4497 raft_consensus.cc:2932] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:06.525266 4091 catalog_manager.cc:5484] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:06.527990 7133 tablet_service.cc:1467] Processing CreateTablet for tablet 92ce2c1ce3e0449bab5490b690721cbf (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=be6eb76edc624f598b8543519c4d4476]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:06.528499 7133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92ce2c1ce3e0449bab5490b690721cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:06.528615 7266 tablet_service.cc:1467] Processing CreateTablet for tablet 92ce2c1ce3e0449bab5490b690721cbf (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=be6eb76edc624f598b8543519c4d4476]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:06.528721 4478 tablet_service.cc:1467] Processing CreateTablet for tablet 61af956bc72b4e3e994a7daf58d1a699 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=be6eb76edc624f598b8543519c4d4476]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:06.529033 7266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92ce2c1ce3e0449bab5490b690721cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:06.529151 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61af956bc72b4e3e994a7daf58d1a699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:06.535645 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 92ce2c1ce3e0449bab5490b690721cbf (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=be6eb76edc624f598b8543519c4d4476]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:06.535696 7400 tablet_service.cc:1467] Processing CreateTablet for tablet 61af956bc72b4e3e994a7daf58d1a699 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=be6eb76edc624f598b8543519c4d4476]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:06.536137 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92ce2c1ce3e0449bab5490b690721cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:06.540315 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61af956bc72b4e3e994a7daf58d1a699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:06.556581 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 61af956bc72b4e3e994a7daf58d1a699 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=be6eb76edc624f598b8543519c4d4476]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:06.557082 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61af956bc72b4e3e994a7daf58d1a699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:06.558841 4091 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:06.567413 7134 tablet_service.cc:1467] Processing CreateTablet for tablet 25f01a01106f47c489e252c928cf28bb (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=be6eb76edc624f598b8543519c4d4476]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:06.568101 7134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f01a01106f47c489e252c928cf28bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:06.571178 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 25f01a01106f47c489e252c928cf28bb (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=be6eb76edc624f598b8543519c4d4476]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:06.571651 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f01a01106f47c489e252c928cf28bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:06.580785 8382 tablet_bootstrap.cc:492] T 92ce2c1ce3e0449bab5490b690721cbf P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:06.583807 4364 raft_consensus.cc:2932] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:06.585588 8382 tablet_bootstrap.cc:654] T 92ce2c1ce3e0449bab5490b690721cbf P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:06.588701 8381 ts_tablet_manager.cc:902] T 644d4d293e8c462297c6419965bd2c89 P ef0a4ada548446b3a531cb39a3c4e38d: Initiating tablet copy from peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:06.590721 8380 tablet_bootstrap.cc:492] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:06.594336 8380 tablet_bootstrap.cc:654] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:06.604250 8378 ts_tablet_manager.cc:902] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97: Initiating tablet copy from peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:06.608623 8345 tablet.cc:1722] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:06.614988 8355 tablet_bootstrap.cc:492] T 9d165c29dabe4ae3be11ed84702c3cb2 P db0742ee3860468590eefef6ae6074fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:06.615557 8355 tablet_bootstrap.cc:492] T 9d165c29dabe4ae3be11ed84702c3cb2 P db0742ee3860468590eefef6ae6074fd: Bootstrap complete.
I20240501 20:10:06.615886 8355 ts_tablet_manager.cc:1367] T 9d165c29dabe4ae3be11ed84702c3cb2 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.116s user 0.051s sys 0.010s
I20240501 20:10:06.617435 8355 raft_consensus.cc:348] T 9d165c29dabe4ae3be11ed84702c3cb2 P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:06.617717 8355 raft_consensus.cc:729] T 9d165c29dabe4ae3be11ed84702c3cb2 P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: LEARNER
I20240501 20:10:06.622313 8380 tablet_bootstrap.cc:492] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:06.624056 8380 ts_tablet_manager.cc:1367] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.000s
I20240501 20:10:06.624212 7266 tablet_service.cc:1467] Processing CreateTablet for tablet 25f01a01106f47c489e252c928cf28bb (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=be6eb76edc624f598b8543519c4d4476]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:06.624708 7266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f01a01106f47c489e252c928cf28bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:06.626672 8348 tablet.cc:1722] T 2c1f95b56c56409990c7aee0f8a9fb8f P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:06.629029 8346 tablet.cc:1722] T 2c1f95b56c56409990c7aee0f8a9fb8f P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:06.628365 8380 raft_consensus.cc:348] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:06.629927 8380 raft_consensus.cc:374] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:06.630542 8380 raft_consensus.cc:729] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:06.631831 8380 consensus_queue.cc:260] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:06.641916 4084 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table_new_name [id=7fff78ff161d4f66bcc8ca9dc1940a47] alter complete (version 3)
I20240501 20:10:06.647142 8380 ts_tablet_manager.cc:1398] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240501 20:10:06.648914 8380 tablet_bootstrap.cc:492] T 25f01a01106f47c489e252c928cf28bb P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:06.651407 8380 tablet_bootstrap.cc:654] T 25f01a01106f47c489e252c928cf28bb P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:06.617949 8355 consensus_queue.cc:260] T 9d165c29dabe4ae3be11ed84702c3cb2 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:06.656482 8355 ts_tablet_manager.cc:1398] T 9d165c29dabe4ae3be11ed84702c3cb2 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.040s user 0.003s sys 0.000s
I20240501 20:10:06.657387 4519 tablet_copy_service.cc:342] P 3e974c03d3c546f9ab6c91e762a5426b: Request end of tablet copy session db0742ee3860468590eefef6ae6074fd-9d165c29dabe4ae3be11ed84702c3cb2 received from {username='slave'} at 127.29.219.70:56941
I20240501 20:10:06.657593 4519 tablet_copy_service.cc:433] P 3e974c03d3c546f9ab6c91e762a5426b: ending tablet copy session db0742ee3860468590eefef6ae6074fd-9d165c29dabe4ae3be11ed84702c3cb2 on tablet 9d165c29dabe4ae3be11ed84702c3cb2 with peer db0742ee3860468590eefef6ae6074fd
I20240501 20:10:06.673521 8395 tablet_bootstrap.cc:492] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:06.674808 8398 ts_tablet_manager.cc:902] T ef7c0a48f11945d3953f3461d06bc184 P b9957d4086bc4a76b18ed9265a5739d5: Initiating tablet copy from peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:06.679821 8395 tablet_bootstrap.cc:654] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:06.685400 8392 ts_tablet_manager.cc:902] T fcff4be357084bf0903cfb1c0f62a866 P ef0a4ada548446b3a531cb39a3c4e38d: Initiating tablet copy from peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:06.686117 8380 tablet_bootstrap.cc:492] T 25f01a01106f47c489e252c928cf28bb P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:06.686348 8380 ts_tablet_manager.cc:1367] T 25f01a01106f47c489e252c928cf28bb P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.000s
I20240501 20:10:06.686767 8393 tablet_bootstrap.cc:492] T 61af956bc72b4e3e994a7daf58d1a699 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:06.687234 8380 raft_consensus.cc:348] T 25f01a01106f47c489e252c928cf28bb P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:06.687549 8380 raft_consensus.cc:374] T 25f01a01106f47c489e252c928cf28bb P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:06.687675 8380 raft_consensus.cc:729] T 25f01a01106f47c489e252c928cf28bb P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:06.688048 8380 consensus_queue.cc:260] T 25f01a01106f47c489e252c928cf28bb P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:06.689594 8380 ts_tablet_manager.cc:1398] T 25f01a01106f47c489e252c928cf28bb P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:10:06.690850 8384 tablet_bootstrap.cc:492] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:06.691169 8378 tablet_copy_client.cc:308] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.66:40689
I20240501 20:10:06.692000 4388 tablet_copy_service.cc:140] P 10b47782558c41ee91f5e164a372049a: Received BeginTabletCopySession request for tablet c66514deec5744da832bc030a582c46a from peer bceeae1c79964acabdae8ff771bf0c97 ({username='slave'} at 127.29.219.71:41437)
I20240501 20:10:06.692190 4388 tablet_copy_service.cc:161] P 10b47782558c41ee91f5e164a372049a: Beginning new tablet copy session on tablet c66514deec5744da832bc030a582c46a from peer bceeae1c79964acabdae8ff771bf0c97 at {username='slave'} at 127.29.219.71:41437: session id = bceeae1c79964acabdae8ff771bf0c97-c66514deec5744da832bc030a582c46a
I20240501 20:10:06.692907 8384 tablet_bootstrap.cc:654] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:06.693181 8395 tablet_bootstrap.cc:492] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:06.693404 8395 ts_tablet_manager.cc:1367] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240501 20:10:06.694535 8395 raft_consensus.cc:348] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:06.695106 4388 tablet_copy_source_session.cc:205] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:06.695283 8395 raft_consensus.cc:374] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:06.695538 8395 raft_consensus.cc:729] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:06.698946 8395 consensus_queue.cc:260] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:06.700973 8395 ts_tablet_manager.cc:1398] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 20:10:06.701822 8071 raft_consensus.cc:482] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:06.702067 8071 raft_consensus.cc:504] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:06.703199 8383 tablet_bootstrap.cc:492] T 61af956bc72b4e3e994a7daf58d1a699 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:06.703341 8071 leader_election.cc:288] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:06.704133 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f01a01106f47c489e252c928cf28bb" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:06.704387 7153 raft_consensus.cc:2454] T 25f01a01106f47c489e252c928cf28bb P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:10:06.704864 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f01a01106f47c489e252c928cf28bb" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:06.705724 8383 tablet_bootstrap.cc:654] T 61af956bc72b4e3e994a7daf58d1a699 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
W20240501 20:10:06.705966 4563 leader_election.cc:341] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:06.706328 4562 leader_election.cc:302] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, db0742ee3860468590eefef6ae6074fd; no voters: bceeae1c79964acabdae8ff771bf0c97
I20240501 20:10:06.706909 8071 raft_consensus.cc:2781] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:06.707048 8071 raft_consensus.cc:482] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:06.707149 8071 raft_consensus.cc:3037] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:06.710479 8393 tablet_bootstrap.cc:654] T 61af956bc72b4e3e994a7daf58d1a699 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:06.711300 8071 raft_consensus.cc:504] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:06.713095 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f01a01106f47c489e252c928cf28bb" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:06.714428 7153 raft_consensus.cc:3037] T 25f01a01106f47c489e252c928cf28bb P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:06.717442 7153 raft_consensus.cc:2454] T 25f01a01106f47c489e252c928cf28bb P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:10:06.717700 8382 tablet_bootstrap.cc:492] T 92ce2c1ce3e0449bab5490b690721cbf P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:06.717867 8382 ts_tablet_manager.cc:1367] T 92ce2c1ce3e0449bab5490b690721cbf P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.137s user 0.019s sys 0.043s
I20240501 20:10:06.718230 4562 leader_election.cc:302] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:06.718823 8314 raft_consensus.cc:2781] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:06.718987 8314 raft_consensus.cc:686] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Becoming Leader. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Running, Role: LEADER
I20240501 20:10:06.718899 8382 raft_consensus.cc:348] T 92ce2c1ce3e0449bab5490b690721cbf P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:06.719162 8382 raft_consensus.cc:374] T 92ce2c1ce3e0449bab5490b690721cbf P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:06.719260 8382 raft_consensus.cc:729] T 92ce2c1ce3e0449bab5490b690721cbf P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:06.719341 8314 consensus_queue.cc:237] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:06.719491 8382 consensus_queue.cc:260] T 92ce2c1ce3e0449bab5490b690721cbf P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:06.720021 8382 ts_tablet_manager.cc:1398] T 92ce2c1ce3e0449bab5490b690721cbf P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:06.721110 8382 tablet_bootstrap.cc:492] T 25f01a01106f47c489e252c928cf28bb P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:06.722693 8382 tablet_bootstrap.cc:654] T 25f01a01106f47c489e252c928cf28bb P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:06.726322 8384 tablet_bootstrap.cc:492] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:10:06.726496 8384 ts_tablet_manager.cc:1367] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.000s
I20240501 20:10:06.727488 8384 raft_consensus.cc:348] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:06.727677 8384 raft_consensus.cc:374] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:06.727768 8384 raft_consensus.cc:729] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:10:06.728061 8384 consensus_queue.cc:260] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:06.728713 8384 ts_tablet_manager.cc:1398] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:06.731233 8355 ts_tablet_manager.cc:902] T 7725fa9f98e9439797a532ed6f2f7dba P db0742ee3860468590eefef6ae6074fd: Initiating tablet copy from peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:06.734287 8366 raft_consensus.cc:482] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:06.734498 8366 raft_consensus.cc:504] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:06.735360 8366 leader_election.cc:288] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:06.736151 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f01a01106f47c489e252c928cf28bb" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
W20240501 20:10:06.736682 4563 leader_election.cc:341] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:06.737018 7421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92ce2c1ce3e0449bab5490b690721cbf" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:06.737687 8071 leader_election.cc:288] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Requested vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
W20240501 20:10:06.737710 7091 leader_election.cc:341] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:06.738076 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92ce2c1ce3e0449bab5490b690721cbf" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:06.738324 7287 raft_consensus.cc:2454] T 92ce2c1ce3e0449bab5490b690721cbf P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:06.738831 7089 leader_election.cc:302] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, db0742ee3860468590eefef6ae6074fd; no voters: b9957d4086bc4a76b18ed9265a5739d5
I20240501 20:10:06.739317 8366 raft_consensus.cc:2781] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:06.739414 8366 raft_consensus.cc:482] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:06.739480 8366 raft_consensus.cc:3037] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:06.743700 8366 raft_consensus.cc:504] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:06.745072 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92ce2c1ce3e0449bab5490b690721cbf" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:06.745342 7287 raft_consensus.cc:3037] T 92ce2c1ce3e0449bab5490b690721cbf P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:06.748458 8382 tablet_bootstrap.cc:492] T 25f01a01106f47c489e252c928cf28bb P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:06.748659 8382 ts_tablet_manager.cc:1367] T 25f01a01106f47c489e252c928cf28bb P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.003s
I20240501 20:10:06.749548 8382 raft_consensus.cc:348] T 25f01a01106f47c489e252c928cf28bb P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:06.750735 8366 leader_election.cc:288] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Requested vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:06.751011 7421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92ce2c1ce3e0449bab5490b690721cbf" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
W20240501 20:10:06.751605 7091 leader_election.cc:341] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:06.749676 7287 raft_consensus.cc:2454] T 92ce2c1ce3e0449bab5490b690721cbf P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:06.752221 7089 leader_election.cc:302] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, db0742ee3860468590eefef6ae6074fd; no voters: b9957d4086bc4a76b18ed9265a5739d5
I20240501 20:10:06.752682 8398 tablet_copy_client.cc:308] T ef7c0a48f11945d3953f3461d06bc184 P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.67:34595
I20240501 20:10:06.752717 8366 raft_consensus.cc:2781] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:06.752928 8366 raft_consensus.cc:686] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Becoming Leader. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Running, Role: LEADER
I20240501 20:10:06.753216 8366 consensus_queue.cc:237] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:06.757781 8383 tablet_bootstrap.cc:492] T 61af956bc72b4e3e994a7daf58d1a699 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:06.757980 8383 ts_tablet_manager.cc:1367] T 61af956bc72b4e3e994a7daf58d1a699 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.001s
I20240501 20:10:06.760231 4083 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:06.761057 4083 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:06.761873 8381 tablet_copy_client.cc:308] T 644d4d293e8c462297c6419965bd2c89 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.67:34595
I20240501 20:10:06.762593 8393 tablet_bootstrap.cc:492] T 61af956bc72b4e3e994a7daf58d1a699 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:06.762840 8393 ts_tablet_manager.cc:1367] T 61af956bc72b4e3e994a7daf58d1a699 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.076s user 0.004s sys 0.001s
I20240501 20:10:06.763743 8393 raft_consensus.cc:348] T 61af956bc72b4e3e994a7daf58d1a699 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:06.763949 8393 raft_consensus.cc:374] T 61af956bc72b4e3e994a7daf58d1a699 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:06.764034 8393 raft_consensus.cc:729] T 61af956bc72b4e3e994a7daf58d1a699 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:06.764299 8393 consensus_queue.cc:260] T 61af956bc72b4e3e994a7daf58d1a699 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:06.765753 8383 raft_consensus.cc:348] T 61af956bc72b4e3e994a7daf58d1a699 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:06.766036 8383 raft_consensus.cc:374] T 61af956bc72b4e3e994a7daf58d1a699 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:06.766145 8383 raft_consensus.cc:729] T 61af956bc72b4e3e994a7daf58d1a699 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:06.749816 8382 raft_consensus.cc:374] T 25f01a01106f47c489e252c928cf28bb P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:06.766932 8382 raft_consensus.cc:729] T 25f01a01106f47c489e252c928cf28bb P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:06.767803 8382 consensus_queue.cc:260] T 25f01a01106f47c489e252c928cf28bb P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:06.769104 8383 consensus_queue.cc:260] T 61af956bc72b4e3e994a7daf58d1a699 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:06.768376 8382 ts_tablet_manager.cc:1398] T 25f01a01106f47c489e252c928cf28bb P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.020s user 0.001s sys 0.002s
I20240501 20:10:06.778281 4083 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 2c1f95b56c56409990c7aee0f8a9fb8f
I20240501 20:10:06.772526 8383 ts_tablet_manager.cc:1398] T 61af956bc72b4e3e994a7daf58d1a699 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240501 20:10:06.779531 8393 ts_tablet_manager.cc:1398] T 61af956bc72b4e3e994a7daf58d1a699 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20240501 20:10:06.781324 7400 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c1f95b56c56409990c7aee0f8a9fb8f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:06 UTC) from {username='slave'} at 127.0.0.1:40182
I20240501 20:10:06.782639 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c1f95b56c56409990c7aee0f8a9fb8f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:06 UTC) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:06.786269 7000 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c1f95b56c56409990c7aee0f8a9fb8f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:06 UTC) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:06.795202 8378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c66514deec5744da832bc030a582c46a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:06.798228 8393 tablet_bootstrap.cc:492] T 92ce2c1ce3e0449bab5490b690721cbf P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:06.802126 8426 tablet_replica.cc:330] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:06.802459 8426 raft_consensus.cc:2229] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:06.803004 8426 raft_consensus.cc:2258] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:06.804193 8393 tablet_bootstrap.cc:654] T 92ce2c1ce3e0449bab5490b690721cbf P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:06.801739 8425 tablet_replica.cc:330] T 2c1f95b56c56409990c7aee0f8a9fb8f P b9957d4086bc4a76b18ed9265a5739d5: stopping tablet replica
I20240501 20:10:06.810894 8425 raft_consensus.cc:2229] T 2c1f95b56c56409990c7aee0f8a9fb8f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:06.811251 8425 raft_consensus.cc:2258] T 2c1f95b56c56409990c7aee0f8a9fb8f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:06.812441 8425 ts_tablet_manager.cc:1875] T 2c1f95b56c56409990c7aee0f8a9fb8f P b9957d4086bc4a76b18ed9265a5739d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:06.817569 8210 raft_consensus.cc:482] T 25f01a01106f47c489e252c928cf28bb P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:06.817773 8210 raft_consensus.cc:504] T 25f01a01106f47c489e252c928cf28bb P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:06.818729 8210 leader_election.cc:288] T 25f01a01106f47c489e252c928cf28bb P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:06.821094 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f01a01106f47c489e252c928cf28bb" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:06.804095 8426 ts_tablet_manager.cc:1875] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:06.822130 4083 catalog_manager.cc:5484] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68). New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:06.823652 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25f01a01106f47c489e252c928cf28bb" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:06.823957 7153 raft_consensus.cc:2379] T 25f01a01106f47c489e252c928cf28bb P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bceeae1c79964acabdae8ff771bf0c97 in current term 1: Already voted for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in this term.
I20240501 20:10:06.824505 7220 leader_election.cc:302] T 25f01a01106f47c489e252c928cf28bb P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97; no voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, db0742ee3860468590eefef6ae6074fd
I20240501 20:10:06.827437 8427 tablet_replica.cc:330] T 2c1f95b56c56409990c7aee0f8a9fb8f P ef0a4ada548446b3a531cb39a3c4e38d: stopping tablet replica
I20240501 20:10:06.827492 8210 raft_consensus.cc:3037] T 25f01a01106f47c489e252c928cf28bb P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:06.827715 8427 raft_consensus.cc:2229] T 2c1f95b56c56409990c7aee0f8a9fb8f P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:06.829665 8427 raft_consensus.cc:2258] T 2c1f95b56c56409990c7aee0f8a9fb8f P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:06.831609 8210 raft_consensus.cc:2726] T 25f01a01106f47c489e252c928cf28bb P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:06.832901 8393 tablet_bootstrap.cc:492] T 92ce2c1ce3e0449bab5490b690721cbf P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:06.833161 8393 ts_tablet_manager.cc:1367] T 92ce2c1ce3e0449bab5490b690721cbf P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.001s
I20240501 20:10:06.836079 8393 raft_consensus.cc:348] T 92ce2c1ce3e0449bab5490b690721cbf P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:06.836395 8393 raft_consensus.cc:374] T 92ce2c1ce3e0449bab5490b690721cbf P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:06.836496 8393 raft_consensus.cc:729] T 92ce2c1ce3e0449bab5490b690721cbf P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:06.836769 8393 consensus_queue.cc:260] T 92ce2c1ce3e0449bab5490b690721cbf P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:06.838721 4084 catalog_manager.cc:5484] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd reported cstate change: term changed from 0 to 1, leader changed from <none> to db0742ee3860468590eefef6ae6074fd (127.29.219.70). New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:06.846148 8425 ts_tablet_manager.cc:1888] T 2c1f95b56c56409990c7aee0f8a9fb8f P b9957d4086bc4a76b18ed9265a5739d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:06.846558 8427 ts_tablet_manager.cc:1875] T 2c1f95b56c56409990c7aee0f8a9fb8f P ef0a4ada548446b3a531cb39a3c4e38d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:06.854749 8392 tablet_copy_client.cc:308] T fcff4be357084bf0903cfb1c0f62a866 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.68:45205
I20240501 20:10:06.850556 8426 ts_tablet_manager.cc:1888] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:06.855809 8426 log.cc:1196] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/2c1f95b56c56409990c7aee0f8a9fb8f
I20240501 20:10:06.856918 4651 tablet_copy_service.cc:140] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Received BeginTabletCopySession request for tablet fcff4be357084bf0903cfb1c0f62a866 from peer ef0a4ada548446b3a531cb39a3c4e38d ({username='slave'} at 127.29.219.69:59565)
I20240501 20:10:06.857195 4651 tablet_copy_service.cc:161] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Beginning new tablet copy session on tablet fcff4be357084bf0903cfb1c0f62a866 from peer ef0a4ada548446b3a531cb39a3c4e38d at {username='slave'} at 127.29.219.69:59565: session id = ef0a4ada548446b3a531cb39a3c4e38d-fcff4be357084bf0903cfb1c0f62a866
I20240501 20:10:06.857685 8426 ts_tablet_manager.cc:1909] T 2c1f95b56c56409990c7aee0f8a9fb8f P 3e974c03d3c546f9ab6c91e762a5426b: Deleting consensus metadata
I20240501 20:10:06.861188 4651 tablet_copy_source_session.cc:205] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:06.861358 8425 log.cc:1196] T 2c1f95b56c56409990c7aee0f8a9fb8f P b9957d4086bc4a76b18ed9265a5739d5: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/wals/2c1f95b56c56409990c7aee0f8a9fb8f
I20240501 20:10:06.861923 8425 ts_tablet_manager.cc:1909] T 2c1f95b56c56409990c7aee0f8a9fb8f P b9957d4086bc4a76b18ed9265a5739d5: Deleting consensus metadata
I20240501 20:10:06.847100 8393 ts_tablet_manager.cc:1398] T 92ce2c1ce3e0449bab5490b690721cbf P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.014s user 0.003s sys 0.001s
I20240501 20:10:06.864333 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet 2c1f95b56c56409990c7aee0f8a9fb8f (table rebalancer_dynamic_table_new_name [id=7fff78ff161d4f66bcc8ca9dc1940a47]) successfully deleted
I20240501 20:10:06.864413 4079 catalog_manager.cc:4830] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): tablet 2c1f95b56c56409990c7aee0f8a9fb8f (table rebalancer_dynamic_table_new_name [id=7fff78ff161d4f66bcc8ca9dc1940a47]) successfully deleted
I20240501 20:10:06.864559 8355 tablet_copy_client.cc:308] T 7725fa9f98e9439797a532ed6f2f7dba P db0742ee3860468590eefef6ae6074fd: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.68:45205
I20240501 20:10:06.864648 8392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcff4be357084bf0903cfb1c0f62a866. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:06.865250 4651 tablet_copy_service.cc:140] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Received BeginTabletCopySession request for tablet 7725fa9f98e9439797a532ed6f2f7dba from peer db0742ee3860468590eefef6ae6074fd ({username='slave'} at 127.29.219.70:45393)
I20240501 20:10:06.865463 4651 tablet_copy_service.cc:161] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Beginning new tablet copy session on tablet 7725fa9f98e9439797a532ed6f2f7dba from peer db0742ee3860468590eefef6ae6074fd at {username='slave'} at 127.29.219.70:45393: session id = db0742ee3860468590eefef6ae6074fd-7725fa9f98e9439797a532ed6f2f7dba
I20240501 20:10:06.869735 8388 ts_tablet_manager.cc:902] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5: Initiating tablet copy from peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:06.869820 4651 tablet_copy_source_session.cc:205] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:06.871089 8355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7725fa9f98e9439797a532ed6f2f7dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:06.875066 8392 tablet_copy_client.cc:783] T fcff4be357084bf0903cfb1c0f62a866 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:06.875506 8392 tablet_copy_client.cc:647] T fcff4be357084bf0903cfb1c0f62a866 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:06.877895 8392 tablet_copy_client.cc:523] T fcff4be357084bf0903cfb1c0f62a866 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:06.882261 8392 tablet_bootstrap.cc:492] T fcff4be357084bf0903cfb1c0f62a866 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:06.883359 8355 tablet_copy_client.cc:783] T 7725fa9f98e9439797a532ed6f2f7dba P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 0 data blocks...
W20240501 20:10:06.883775 7356 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.68:45205, user_credentials={real_user=slave}} blocked reactor thread for 52910.4us
I20240501 20:10:06.886926 8355 tablet_copy_client.cc:647] T 7725fa9f98e9439797a532ed6f2f7dba P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:06.891420 4519 tablet_copy_service.cc:140] P 3e974c03d3c546f9ab6c91e762a5426b: Received BeginTabletCopySession request for tablet 644d4d293e8c462297c6419965bd2c89 from peer ef0a4ada548446b3a531cb39a3c4e38d ({username='slave'} at 127.29.219.69:60767)
I20240501 20:10:06.891613 4519 tablet_copy_service.cc:161] P 3e974c03d3c546f9ab6c91e762a5426b: Beginning new tablet copy session on tablet 644d4d293e8c462297c6419965bd2c89 from peer ef0a4ada548446b3a531cb39a3c4e38d at {username='slave'} at 127.29.219.69:60767: session id = ef0a4ada548446b3a531cb39a3c4e38d-644d4d293e8c462297c6419965bd2c89
I20240501 20:10:06.896183 8355 tablet_copy_client.cc:523] T 7725fa9f98e9439797a532ed6f2f7dba P db0742ee3860468590eefef6ae6074fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:06.898674 4519 tablet_copy_source_session.cc:205] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:06.899648 4084 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:06.900060 8355 tablet_bootstrap.cc:492] T 7725fa9f98e9439797a532ed6f2f7dba P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:06.909308 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 38d1f8cee44d4a3a9c378733e3b146cf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2fa59113bdc54d6aabf88f36456b3840]), partition=
I20240501 20:10:06.909703 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d1f8cee44d4a3a9c378733e3b146cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:06.910423 7266 tablet_service.cc:1467] Processing CreateTablet for tablet 38d1f8cee44d4a3a9c378733e3b146cf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2fa59113bdc54d6aabf88f36456b3840]), partition=
I20240501 20:10:06.910867 7266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d1f8cee44d4a3a9c378733e3b146cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:06.913010 7400 tablet_service.cc:1467] Processing CreateTablet for tablet 38d1f8cee44d4a3a9c378733e3b146cf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2fa59113bdc54d6aabf88f36456b3840]), partition=
I20240501 20:10:06.913383 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d1f8cee44d4a3a9c378733e3b146cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:06.914696 8153 raft_consensus.cc:482] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:06.914753 7155 raft_consensus.cc:1203] T 9d165c29dabe4ae3be11ed84702c3cb2 P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:06.914865 8153 raft_consensus.cc:504] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:06.915660 8153 leader_election.cc:288] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:06.916332 7421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61af956bc72b4e3e994a7daf58d1a699" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:06.916559 7421 raft_consensus.cc:2454] T 61af956bc72b4e3e994a7daf58d1a699 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:10:06.917307 4435 leader_election.cc:302] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, b9957d4086bc4a76b18ed9265a5739d5; no voters:
I20240501 20:10:06.918023 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61af956bc72b4e3e994a7daf58d1a699" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:06.918274 4234 raft_consensus.cc:2454] T 61af956bc72b4e3e994a7daf58d1a699 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:10:06.919090 8382 tablet_bootstrap.cc:492] T 38d1f8cee44d4a3a9c378733e3b146cf P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:06.920514 8438 ts_tablet_manager.cc:902] T 41d6e6dce35e4d0da594f147a7f9bfc4 P db0742ee3860468590eefef6ae6074fd: Initiating tablet copy from peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:06.920746 8382 tablet_bootstrap.cc:654] T 38d1f8cee44d4a3a9c378733e3b146cf P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:06.922796 8381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 644d4d293e8c462297c6419965bd2c89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:06.925025 8250 raft_consensus.cc:2781] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:06.925123 8250 raft_consensus.cc:482] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:06.925206 8250 raft_consensus.cc:3037] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:06.928346 8250 raft_consensus.cc:504] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:06.929502 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61af956bc72b4e3e994a7daf58d1a699" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:06.929749 7420 raft_consensus.cc:3037] T 61af956bc72b4e3e994a7daf58d1a699 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:06.932663 7420 raft_consensus.cc:2454] T 61af956bc72b4e3e994a7daf58d1a699 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:10:06.934324 4435 leader_election.cc:302] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, b9957d4086bc4a76b18ed9265a5739d5; no voters:
I20240501 20:10:06.934882 8251 raft_consensus.cc:2781] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:06.935388 8250 leader_election.cc:288] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Requested vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:06.936277 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61af956bc72b4e3e994a7daf58d1a699" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:10:06.936555 4234 raft_consensus.cc:3037] T 61af956bc72b4e3e994a7daf58d1a699 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:06.938710 8251 raft_consensus.cc:686] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:10:06.939070 8251 consensus_queue.cc:237] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:06.944015 4234 raft_consensus.cc:2454] T 61af956bc72b4e3e994a7daf58d1a699 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:10:06.944478 8393 tablet_bootstrap.cc:492] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:06.946164 8393 tablet_bootstrap.cc:654] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:06.948827 8381 tablet_copy_client.cc:783] T 644d4d293e8c462297c6419965bd2c89 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:06.949651 8381 tablet_copy_client.cc:647] T 644d4d293e8c462297c6419965bd2c89 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:06.955412 8395 tablet_bootstrap.cc:492] T 38d1f8cee44d4a3a9c378733e3b146cf P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:06.958117 8395 tablet_bootstrap.cc:654] T 38d1f8cee44d4a3a9c378733e3b146cf P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:06.972648 4627 consensus_queue.cc:237] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:06.980247 8393 tablet_bootstrap.cc:492] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:06.981506 8395 tablet_bootstrap.cc:492] T 38d1f8cee44d4a3a9c378733e3b146cf P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:06.982326 8395 ts_tablet_manager.cc:1367] T 38d1f8cee44d4a3a9c378733e3b146cf P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.008s
I20240501 20:10:06.982640 8393 ts_tablet_manager.cc:1367] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.010s
I20240501 20:10:06.983698 8395 raft_consensus.cc:348] T 38d1f8cee44d4a3a9c378733e3b146cf P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:06.984931 8393 raft_consensus.cc:348] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:06.985308 8395 raft_consensus.cc:374] T 38d1f8cee44d4a3a9c378733e3b146cf P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:06.985461 8395 raft_consensus.cc:729] T 38d1f8cee44d4a3a9c378733e3b146cf P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:06.985601 8393 raft_consensus.cc:374] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:06.985917 8395 consensus_queue.cc:260] T 38d1f8cee44d4a3a9c378733e3b146cf P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:06.986243 8393 raft_consensus.cc:729] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:06.986989 8393 consensus_queue.cc:260] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:06.987766 8395 ts_tablet_manager.cc:1398] T 38d1f8cee44d4a3a9c378733e3b146cf P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 20:10:06.988893 8382 tablet_bootstrap.cc:492] T 38d1f8cee44d4a3a9c378733e3b146cf P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:06.989384 8382 ts_tablet_manager.cc:1367] T 38d1f8cee44d4a3a9c378733e3b146cf P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.022s
I20240501 20:10:06.989439 8393 ts_tablet_manager.cc:1398] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:10:06.990450 8382 raft_consensus.cc:348] T 38d1f8cee44d4a3a9c378733e3b146cf P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:06.990695 8382 raft_consensus.cc:374] T 38d1f8cee44d4a3a9c378733e3b146cf P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:06.991906 8382 raft_consensus.cc:729] T 38d1f8cee44d4a3a9c378733e3b146cf P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:06.992451 8382 consensus_queue.cc:260] T 38d1f8cee44d4a3a9c378733e3b146cf P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:06.993417 8382 ts_tablet_manager.cc:1398] T 38d1f8cee44d4a3a9c378733e3b146cf P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:10:07.006657 8449 ts_tablet_manager.cc:902] T bb85d7825c564339bd554deeded45cec P 1d34e18a1d394d3f91c9f50d1c7bea4e: Initiating tablet copy from peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:07.018714 4083 catalog_manager.cc:5484] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:07.030526 8028 raft_consensus.cc:482] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:07.030819 8028 raft_consensus.cc:504] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:07.031718 8028 leader_election.cc:288] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:07.032591 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d1f8cee44d4a3a9c378733e3b146cf" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:07.032872 7284 raft_consensus.cc:2454] T 38d1f8cee44d4a3a9c378733e3b146cf P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:10:07.033684 7354 leader_election.cc:302] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:07.034202 8449 tablet_copy_client.cc:308] T bb85d7825c564339bd554deeded45cec P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.70:36321
I20240501 20:10:07.034449 8028 raft_consensus.cc:2781] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:07.034555 8028 raft_consensus.cc:482] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:07.035252 8028 raft_consensus.cc:3037] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:07.035553 7175 tablet_copy_service.cc:140] P db0742ee3860468590eefef6ae6074fd: Received BeginTabletCopySession request for tablet bb85d7825c564339bd554deeded45cec from peer 1d34e18a1d394d3f91c9f50d1c7bea4e ({username='slave'} at 127.29.219.68:44315)
I20240501 20:10:07.035729 7175 tablet_copy_service.cc:161] P db0742ee3860468590eefef6ae6074fd: Beginning new tablet copy session on tablet bb85d7825c564339bd554deeded45cec from peer 1d34e18a1d394d3f91c9f50d1c7bea4e at {username='slave'} at 127.29.219.68:44315: session id = 1d34e18a1d394d3f91c9f50d1c7bea4e-bb85d7825c564339bd554deeded45cec
I20240501 20:10:07.038414 7175 tablet_copy_source_session.cc:205] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:07.039426 8028 raft_consensus.cc:504] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:07.041232 8449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb85d7825c564339bd554deeded45cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:07.044204 8355 tablet_bootstrap.cc:492] T 7725fa9f98e9439797a532ed6f2f7dba P db0742ee3860468590eefef6ae6074fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:07.044770 8355 tablet_bootstrap.cc:492] T 7725fa9f98e9439797a532ed6f2f7dba P db0742ee3860468590eefef6ae6074fd: Bootstrap complete.
I20240501 20:10:07.045133 8355 ts_tablet_manager.cc:1367] T 7725fa9f98e9439797a532ed6f2f7dba P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.145s user 0.058s sys 0.002s
I20240501 20:10:07.046258 8355 raft_consensus.cc:348] T 7725fa9f98e9439797a532ed6f2f7dba P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:07.046640 8355 raft_consensus.cc:729] T 7725fa9f98e9439797a532ed6f2f7dba P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: LEARNER
I20240501 20:10:07.046911 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d1f8cee44d4a3a9c378733e3b146cf" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:07.046949 8355 consensus_queue.cc:260] T 7725fa9f98e9439797a532ed6f2f7dba P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:07.047189 7284 raft_consensus.cc:3037] T 38d1f8cee44d4a3a9c378733e3b146cf P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:07.049571 8449 tablet_copy_client.cc:783] T bb85d7825c564339bd554deeded45cec P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:07.049912 8449 tablet_copy_client.cc:647] T bb85d7825c564339bd554deeded45cec P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:07.051054 8355 ts_tablet_manager.cc:1398] T 7725fa9f98e9439797a532ed6f2f7dba P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:10:07.058777 8449 tablet_copy_client.cc:523] T bb85d7825c564339bd554deeded45cec P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:07.059145 8028 leader_election.cc:288] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Requested vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:07.060060 7284 raft_consensus.cc:2454] T 38d1f8cee44d4a3a9c378733e3b146cf P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:07.060297 4651 tablet_copy_service.cc:342] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Request end of tablet copy session db0742ee3860468590eefef6ae6074fd-7725fa9f98e9439797a532ed6f2f7dba received from {username='slave'} at 127.29.219.70:45393
I20240501 20:10:07.060461 4651 tablet_copy_service.cc:433] P 1d34e18a1d394d3f91c9f50d1c7bea4e: ending tablet copy session db0742ee3860468590eefef6ae6074fd-7725fa9f98e9439797a532ed6f2f7dba on tablet 7725fa9f98e9439797a532ed6f2f7dba with peer db0742ee3860468590eefef6ae6074fd
I20240501 20:10:07.060700 7354 leader_election.cc:302] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:07.063042 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d1f8cee44d4a3a9c378733e3b146cf" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:07.063341 4630 raft_consensus.cc:2454] T 38d1f8cee44d4a3a9c378733e3b146cf P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:10:07.063396 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d1f8cee44d4a3a9c378733e3b146cf" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:10:07.063602 8449 tablet_bootstrap.cc:492] T bb85d7825c564339bd554deeded45cec P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:07.063675 4631 raft_consensus.cc:3037] T 38d1f8cee44d4a3a9c378733e3b146cf P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:07.067600 4631 raft_consensus.cc:2454] T 38d1f8cee44d4a3a9c378733e3b146cf P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:07.068632 8028 raft_consensus.cc:2781] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:07.068899 8028 raft_consensus.cc:686] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Becoming Leader. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Running, Role: LEADER
I20240501 20:10:07.069250 8028 consensus_queue.cc:237] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:07.074921 8381 tablet_copy_client.cc:523] T 644d4d293e8c462297c6419965bd2c89 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:07.079468 8449 tablet_bootstrap.cc:492] T bb85d7825c564339bd554deeded45cec P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:07.080061 8449 tablet_bootstrap.cc:492] T bb85d7825c564339bd554deeded45cec P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap complete.
I20240501 20:10:07.080192 8381 tablet_bootstrap.cc:492] T 644d4d293e8c462297c6419965bd2c89 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:07.080346 8449 ts_tablet_manager.cc:1367] T bb85d7825c564339bd554deeded45cec P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.004s
I20240501 20:10:07.081936 8449 raft_consensus.cc:348] T bb85d7825c564339bd554deeded45cec P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:07.084455 8449 raft_consensus.cc:729] T bb85d7825c564339bd554deeded45cec P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: LEARNER
I20240501 20:10:07.084755 8449 consensus_queue.cc:260] T bb85d7825c564339bd554deeded45cec P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:07.085341 8449 ts_tablet_manager.cc:1398] T bb85d7825c564339bd554deeded45cec P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:10:07.086341 7175 tablet_copy_service.cc:342] P db0742ee3860468590eefef6ae6074fd: Request end of tablet copy session 1d34e18a1d394d3f91c9f50d1c7bea4e-bb85d7825c564339bd554deeded45cec received from {username='slave'} at 127.29.219.68:44315
I20240501 20:10:07.086512 7175 tablet_copy_service.cc:433] P db0742ee3860468590eefef6ae6074fd: ending tablet copy session 1d34e18a1d394d3f91c9f50d1c7bea4e-bb85d7825c564339bd554deeded45cec on tablet bb85d7825c564339bd554deeded45cec with peer 1d34e18a1d394d3f91c9f50d1c7bea4e
W20240501 20:10:07.099061 8349 log.cc:924] Time spent T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20240501 20:10:07.105859 4498 raft_consensus.cc:1261] T 1c6c89fa7e6f485e889a5197da4bd135 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:07.106484 4564 consensus_peers.cc:485] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c6c89fa7e6f485e889a5197da4bd135. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:07.106590 8374 consensus_queue.cc:1035] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:07.109086 4234 raft_consensus.cc:1261] T 1c6c89fa7e6f485e889a5197da4bd135 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:07.111507 8314 consensus_queue.cc:1035] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:07.101501 8378 tablet_copy_client.cc:783] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:07.112989 8378 tablet_copy_client.cc:647] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:07.115921 4084 catalog_manager.cc:5484] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72). New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:07.116281 8427 ts_tablet_manager.cc:1888] T 2c1f95b56c56409990c7aee0f8a9fb8f P ef0a4ada548446b3a531cb39a3c4e38d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:07.116425 8427 log.cc:1196] T 2c1f95b56c56409990c7aee0f8a9fb8f P ef0a4ada548446b3a531cb39a3c4e38d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/wals/2c1f95b56c56409990c7aee0f8a9fb8f
I20240501 20:10:07.116415 8378 tablet_copy_client.cc:523] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:07.117018 8427 ts_tablet_manager.cc:1909] T 2c1f95b56c56409990c7aee0f8a9fb8f P ef0a4ada548446b3a531cb39a3c4e38d: Deleting consensus metadata
I20240501 20:10:07.122445 4519 tablet_copy_service.cc:140] P 3e974c03d3c546f9ab6c91e762a5426b: Received BeginTabletCopySession request for tablet ef7c0a48f11945d3953f3461d06bc184 from peer b9957d4086bc4a76b18ed9265a5739d5 ({username='slave'} at 127.29.219.72:58885)
I20240501 20:10:07.122646 4519 tablet_copy_service.cc:161] P 3e974c03d3c546f9ab6c91e762a5426b: Beginning new tablet copy session on tablet ef7c0a48f11945d3953f3461d06bc184 from peer b9957d4086bc4a76b18ed9265a5739d5 at {username='slave'} at 127.29.219.72:58885: session id = b9957d4086bc4a76b18ed9265a5739d5-ef7c0a48f11945d3953f3461d06bc184
I20240501 20:10:07.125577 4519 tablet_copy_source_session.cc:205] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:07.127743 8398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef7c0a48f11945d3953f3461d06bc184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:07.128407 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet 2c1f95b56c56409990c7aee0f8a9fb8f (table rebalancer_dynamic_table_new_name [id=7fff78ff161d4f66bcc8ca9dc1940a47]) successfully deleted
I20240501 20:10:07.131688 8388 tablet_copy_client.cc:308] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.65:42941
I20240501 20:10:07.132542 4254 tablet_copy_service.cc:140] P dca57823003e4729b769ca12367d7530: Received BeginTabletCopySession request for tablet d690199665174c7d9bdbd54d2af4ac70 from peer b9957d4086bc4a76b18ed9265a5739d5 ({username='slave'} at 127.29.219.72:48035)
I20240501 20:10:07.132726 4254 tablet_copy_service.cc:161] P dca57823003e4729b769ca12367d7530: Beginning new tablet copy session on tablet d690199665174c7d9bdbd54d2af4ac70 from peer b9957d4086bc4a76b18ed9265a5739d5 at {username='slave'} at 127.29.219.72:48035: session id = b9957d4086bc4a76b18ed9265a5739d5-d690199665174c7d9bdbd54d2af4ac70
I20240501 20:10:07.136022 4254 tablet_copy_source_session.cc:205] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:07.136371 8398 tablet_copy_client.cc:783] T ef7c0a48f11945d3953f3461d06bc184 P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:07.136689 8398 tablet_copy_client.cc:647] T ef7c0a48f11945d3953f3461d06bc184 P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:07.138667 8314 raft_consensus.cc:2932] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:07.143003 8378 tablet_bootstrap.cc:492] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:07.147758 8374 consensus_queue.cc:1035] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:07.148799 4498 raft_consensus.cc:2932] T 1c6c89fa7e6f485e889a5197da4bd135 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:07.151912 4234 raft_consensus.cc:2932] T 1c6c89fa7e6f485e889a5197da4bd135 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:07.160646 8388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d690199665174c7d9bdbd54d2af4ac70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:07.165542 4087 catalog_manager.cc:5484] T 1c6c89fa7e6f485e889a5197da4bd135 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:07.171032 8398 tablet_copy_client.cc:523] T ef7c0a48f11945d3953f3461d06bc184 P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:07.175787 8398 tablet_bootstrap.cc:492] T ef7c0a48f11945d3953f3461d06bc184 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:07.187131 8438 tablet_copy_client.cc:308] T 41d6e6dce35e4d0da594f147a7f9bfc4 P db0742ee3860468590eefef6ae6074fd: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.68:45205
I20240501 20:10:07.187958 4651 tablet_copy_service.cc:140] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Received BeginTabletCopySession request for tablet 41d6e6dce35e4d0da594f147a7f9bfc4 from peer db0742ee3860468590eefef6ae6074fd ({username='slave'} at 127.29.219.70:45393)
I20240501 20:10:07.188122 4651 tablet_copy_service.cc:161] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Beginning new tablet copy session on tablet 41d6e6dce35e4d0da594f147a7f9bfc4 from peer db0742ee3860468590eefef6ae6074fd at {username='slave'} at 127.29.219.70:45393: session id = db0742ee3860468590eefef6ae6074fd-41d6e6dce35e4d0da594f147a7f9bfc4
I20240501 20:10:07.190310 8388 tablet_copy_client.cc:783] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:07.193971 4651 tablet_copy_source_session.cc:205] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:07.190588 8388 tablet_copy_client.cc:647] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:07.198041 8438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41d6e6dce35e4d0da594f147a7f9bfc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:07.196446 8388 tablet_copy_client.cc:523] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:07.199716 8071 consensus_queue.cc:1035] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:07.204176 8388 tablet_bootstrap.cc:492] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:07.224422 7420 raft_consensus.cc:3037] T 92ce2c1ce3e0449bab5490b690721cbf P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:07.225790 8438 tablet_copy_client.cc:783] T 41d6e6dce35e4d0da594f147a7f9bfc4 P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:07.226192 8438 tablet_copy_client.cc:647] T 41d6e6dce35e4d0da594f147a7f9bfc4 P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:07.233084 8366 consensus_queue.cc:1035] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:07.233269 4084 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:07.236925 7153 raft_consensus.cc:1203] T 7725fa9f98e9439797a532ed6f2f7dba P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:07.237623 8438 tablet_copy_client.cc:523] T 41d6e6dce35e4d0da594f147a7f9bfc4 P db0742ee3860468590eefef6ae6074fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:07.275146 4627 raft_consensus.cc:1261] T 38d1f8cee44d4a3a9c378733e3b146cf P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:07.275921 8430 consensus_queue.cc:1035] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:07.277756 7285 raft_consensus.cc:1261] T 38d1f8cee44d4a3a9c378733e3b146cf P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:07.279238 8430 consensus_queue.cc:1035] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:07.304744 4087 catalog_manager.cc:5576] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205) does not have the latest schema for tablet 38d1f8cee44d4a3a9c378733e3b146cf (table rebalancer_dynamic_table [id=2fa59113bdc54d6aabf88f36456b3840]). Expected version 1 got 0
I20240501 20:10:07.311771 8398 tablet_bootstrap.cc:492] T ef7c0a48f11945d3953f3461d06bc184 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:07.312489 8398 tablet_bootstrap.cc:492] T ef7c0a48f11945d3953f3461d06bc184 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap complete.
I20240501 20:10:07.312841 8398 ts_tablet_manager.cc:1367] T ef7c0a48f11945d3953f3461d06bc184 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.137s user 0.045s sys 0.012s
I20240501 20:10:07.313752 8398 raft_consensus.cc:348] T ef7c0a48f11945d3953f3461d06bc184 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:07.313794 7850 consensus_queue.cc:1035] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:07.315905 8398 raft_consensus.cc:729] T ef7c0a48f11945d3953f3461d06bc184 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: LEARNER
I20240501 20:10:07.316248 8398 consensus_queue.cc:260] T ef7c0a48f11945d3953f3461d06bc184 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:07.317950 8398 ts_tablet_manager.cc:1398] T ef7c0a48f11945d3953f3461d06bc184 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 20:10:07.319744 4087 catalog_manager.cc:5576] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205) does not have the latest schema for tablet 38d1f8cee44d4a3a9c378733e3b146cf (table rebalancer_dynamic_table [id=2fa59113bdc54d6aabf88f36456b3840]). Expected version 1 got 0
I20240501 20:10:07.320324 8479 tablet.cc:1722] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:07.321574 4519 tablet_copy_service.cc:342] P 3e974c03d3c546f9ab6c91e762a5426b: Request end of tablet copy session b9957d4086bc4a76b18ed9265a5739d5-ef7c0a48f11945d3953f3461d06bc184 received from {username='slave'} at 127.29.219.72:58885
I20240501 20:10:07.321733 4519 tablet_copy_service.cc:433] P 3e974c03d3c546f9ab6c91e762a5426b: ending tablet copy session b9957d4086bc4a76b18ed9265a5739d5-ef7c0a48f11945d3953f3461d06bc184 on tablet ef7c0a48f11945d3953f3461d06bc184 with peer b9957d4086bc4a76b18ed9265a5739d5
I20240501 20:10:07.328811 4084 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=2fa59113bdc54d6aabf88f36456b3840] alter complete (version 1)
I20240501 20:10:07.332295 8479 tablet.cc:1620] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:07.348603 8438 tablet_bootstrap.cc:492] T 41d6e6dce35e4d0da594f147a7f9bfc4 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:07.351851 8392 tablet_bootstrap.cc:492] T fcff4be357084bf0903cfb1c0f62a866 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:07.352322 8392 tablet_bootstrap.cc:492] T fcff4be357084bf0903cfb1c0f62a866 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap complete.
I20240501 20:10:07.352597 8392 ts_tablet_manager.cc:1367] T fcff4be357084bf0903cfb1c0f62a866 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.470s user 0.071s sys 0.075s
I20240501 20:10:07.353386 8392 raft_consensus.cc:348] T fcff4be357084bf0903cfb1c0f62a866 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:07.353742 8392 raft_consensus.cc:729] T fcff4be357084bf0903cfb1c0f62a866 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: LEARNER
I20240501 20:10:07.353961 8392 consensus_queue.cc:260] T fcff4be357084bf0903cfb1c0f62a866 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:07.357434 8388 tablet_bootstrap.cc:492] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:07.357971 8388 tablet_bootstrap.cc:492] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap complete.
I20240501 20:10:07.358304 8388 ts_tablet_manager.cc:1367] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.154s user 0.052s sys 0.004s
I20240501 20:10:07.362175 8392 ts_tablet_manager.cc:1398] T fcff4be357084bf0903cfb1c0f62a866 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20240501 20:10:07.363096 4651 tablet_copy_service.cc:342] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Request end of tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-fcff4be357084bf0903cfb1c0f62a866 received from {username='slave'} at 127.29.219.69:59565
I20240501 20:10:07.363265 4651 tablet_copy_service.cc:433] P 1d34e18a1d394d3f91c9f50d1c7bea4e: ending tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-fcff4be357084bf0903cfb1c0f62a866 on tablet fcff4be357084bf0903cfb1c0f62a866 with peer ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:07.363485 8388 raft_consensus.cc:348] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:07.363874 8388 raft_consensus.cc:729] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: LEARNER
I20240501 20:10:07.364136 8388 consensus_queue.cc:260] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:07.366111 8388 ts_tablet_manager.cc:1398] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:10:07.366886 4254 tablet_copy_service.cc:342] P dca57823003e4729b769ca12367d7530: Request end of tablet copy session b9957d4086bc4a76b18ed9265a5739d5-d690199665174c7d9bdbd54d2af4ac70 received from {username='slave'} at 127.29.219.72:48035
I20240501 20:10:07.367028 4254 tablet_copy_service.cc:433] P dca57823003e4729b769ca12367d7530: ending tablet copy session b9957d4086bc4a76b18ed9265a5739d5-d690199665174c7d9bdbd54d2af4ac70 on tablet d690199665174c7d9bdbd54d2af4ac70 with peer b9957d4086bc4a76b18ed9265a5739d5
I20240501 20:10:07.367980 8483 tablet.cc:1722] T 38d1f8cee44d4a3a9c378733e3b146cf P 1d34e18a1d394d3f91c9f50d1c7bea4e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:07.373234 8483 tablet.cc:1620] T 38d1f8cee44d4a3a9c378733e3b146cf P 1d34e18a1d394d3f91c9f50d1c7bea4e: MemRowSet was empty: no flush needed.
I20240501 20:10:07.374929 8251 raft_consensus.cc:1050] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b: attempting to promote NON_VOTER db0742ee3860468590eefef6ae6074fd to VOTER
I20240501 20:10:07.375798 8251 consensus_queue.cc:237] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:07.380627 8381 tablet_bootstrap.cc:492] T 644d4d293e8c462297c6419965bd2c89 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:07.381119 8381 tablet_bootstrap.cc:492] T 644d4d293e8c462297c6419965bd2c89 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap complete.
I20240501 20:10:07.381395 8381 ts_tablet_manager.cc:1367] T 644d4d293e8c462297c6419965bd2c89 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.301s user 0.039s sys 0.020s
I20240501 20:10:07.382165 8381 raft_consensus.cc:348] T 644d4d293e8c462297c6419965bd2c89 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:07.382319 8378 tablet_bootstrap.cc:492] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:07.382373 8381 raft_consensus.cc:729] T 644d4d293e8c462297c6419965bd2c89 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: LEARNER
I20240501 20:10:07.382583 8381 consensus_queue.cc:260] T 644d4d293e8c462297c6419965bd2c89 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:07.382786 8378 tablet_bootstrap.cc:492] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97: Bootstrap complete.
I20240501 20:10:07.383155 8378 ts_tablet_manager.cc:1367] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.240s user 0.057s sys 0.013s
I20240501 20:10:07.387046 8486 tablet.cc:1722] T 38d1f8cee44d4a3a9c378733e3b146cf P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:07.387799 8486 tablet.cc:1620] T 38d1f8cee44d4a3a9c378733e3b146cf P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:07.389338 8381 ts_tablet_manager.cc:1398] T 644d4d293e8c462297c6419965bd2c89 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20240501 20:10:07.389585 4084 catalog_manager.cc:5576] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447) does not have the latest schema for tablet 38d1f8cee44d4a3a9c378733e3b146cf (table rebalancer_dynamic_table [id=2fa59113bdc54d6aabf88f36456b3840]). Expected version 1 got 0
I20240501 20:10:07.391678 4519 tablet_copy_service.cc:342] P 3e974c03d3c546f9ab6c91e762a5426b: Request end of tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-644d4d293e8c462297c6419965bd2c89 received from {username='slave'} at 127.29.219.69:60767
I20240501 20:10:07.391831 4519 tablet_copy_service.cc:433] P 3e974c03d3c546f9ab6c91e762a5426b: ending tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-644d4d293e8c462297c6419965bd2c89 on tablet 644d4d293e8c462297c6419965bd2c89 with peer ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:07.399864 7020 raft_consensus.cc:1203] T 644d4d293e8c462297c6419965bd2c89 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:07.404268 4627 raft_consensus.cc:1261] T 9d165c29dabe4ae3be11ed84702c3cb2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:07.405354 8251 consensus_queue.cc:1035] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:07.405946 4365 raft_consensus.cc:1261] T 9d165c29dabe4ae3be11ed84702c3cb2 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:07.406505 8443 consensus_queue.cc:1035] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:07.407788 7155 raft_consensus.cc:1261] T 9d165c29dabe4ae3be11ed84702c3cb2 P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:07.408529 8443 consensus_queue.cc:1035] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:07.414767 8378 raft_consensus.cc:348] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:07.415236 8378 raft_consensus.cc:729] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: LEARNER
I20240501 20:10:07.415608 8378 consensus_queue.cc:260] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:07.419322 8378 ts_tablet_manager.cc:1398] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20240501 20:10:07.420145 4388 tablet_copy_service.cc:342] P 10b47782558c41ee91f5e164a372049a: Request end of tablet copy session bceeae1c79964acabdae8ff771bf0c97-c66514deec5744da832bc030a582c46a received from {username='slave'} at 127.29.219.71:41437
I20240501 20:10:07.420331 4388 tablet_copy_service.cc:433] P 10b47782558c41ee91f5e164a372049a: ending tablet copy session bceeae1c79964acabdae8ff771bf0c97-c66514deec5744da832bc030a582c46a on tablet c66514deec5744da832bc030a582c46a with peer bceeae1c79964acabdae8ff771bf0c97
I20240501 20:10:07.425388 4630 raft_consensus.cc:1203] T bb85d7825c564339bd554deeded45cec P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:07.429636 8443 raft_consensus.cc:2932] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:07.433682 8349 tablet_bootstrap.cc:492] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:07.434327 7287 raft_consensus.cc:1203] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:07.434370 4627 raft_consensus.cc:2932] T 9d165c29dabe4ae3be11ed84702c3cb2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:07.439942 4087 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:07.440222 8349 tablet_bootstrap.cc:492] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap complete.
I20240501 20:10:07.440590 8349 ts_tablet_manager.cc:1367] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.952s user 0.058s sys 0.010s
I20240501 20:10:07.441653 8349 raft_consensus.cc:348] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:07.442431 8349 raft_consensus.cc:729] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: LEARNER
I20240501 20:10:07.442786 8349 consensus_queue.cc:260] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:07.445346 7155 raft_consensus.cc:2932] T 9d165c29dabe4ae3be11ed84702c3cb2 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:07.445889 8349 ts_tablet_manager.cc:1398] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:10:07.443428 4083 catalog_manager.cc:5484] T 9d165c29dabe4ae3be11ed84702c3cb2 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:07.446893 4388 tablet_copy_service.cc:342] P 10b47782558c41ee91f5e164a372049a: Request end of tablet copy session bceeae1c79964acabdae8ff771bf0c97-f40b062e103d4e54a913fa0f4111f623 received from {username='slave'} at 127.29.219.71:41437
I20240501 20:10:07.447074 4388 tablet_copy_service.cc:433] P 10b47782558c41ee91f5e164a372049a: ending tablet copy session bceeae1c79964acabdae8ff771bf0c97-f40b062e103d4e54a913fa0f4111f623 on tablet f40b062e103d4e54a913fa0f4111f623 with peer bceeae1c79964acabdae8ff771bf0c97
I20240501 20:10:07.448383 8443 consensus_queue.cc:1035] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:07.448132 4365 raft_consensus.cc:2932] T 9d165c29dabe4ae3be11ed84702c3cb2 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:07.448722 4498 consensus_queue.cc:237] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:07.485337 8479 tablet.cc:1722] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:07.485878 8479 tablet.cc:1620] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:07.487610 7020 raft_consensus.cc:1203] T fcff4be357084bf0903cfb1c0f62a866 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:07.489502 8486 tablet.cc:1722] T 38d1f8cee44d4a3a9c378733e3b146cf P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:07.489981 8486 tablet.cc:1620] T 38d1f8cee44d4a3a9c378733e3b146cf P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:07.506834 4084 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=2fa59113bdc54d6aabf88f36456b3840] alter complete (version 2)
I20240501 20:10:07.507174 8438 tablet_bootstrap.cc:492] T 41d6e6dce35e4d0da594f147a7f9bfc4 P db0742ee3860468590eefef6ae6074fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:07.507885 8438 tablet_bootstrap.cc:492] T 41d6e6dce35e4d0da594f147a7f9bfc4 P db0742ee3860468590eefef6ae6074fd: Bootstrap complete.
I20240501 20:10:07.508199 8438 ts_tablet_manager.cc:1367] T 41d6e6dce35e4d0da594f147a7f9bfc4 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.160s user 0.041s sys 0.010s
I20240501 20:10:07.508224 8483 tablet.cc:1722] T 38d1f8cee44d4a3a9c378733e3b146cf P 1d34e18a1d394d3f91c9f50d1c7bea4e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:07.508723 8483 tablet.cc:1620] T 38d1f8cee44d4a3a9c378733e3b146cf P 1d34e18a1d394d3f91c9f50d1c7bea4e: MemRowSet was empty: no flush needed.
I20240501 20:10:07.509080 8438 raft_consensus.cc:348] T 41d6e6dce35e4d0da594f147a7f9bfc4 P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:07.509336 8438 raft_consensus.cc:729] T 41d6e6dce35e4d0da594f147a7f9bfc4 P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: LEARNER
I20240501 20:10:07.509552 8438 consensus_queue.cc:260] T 41d6e6dce35e4d0da594f147a7f9bfc4 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:07.510893 8438 ts_tablet_manager.cc:1398] T 41d6e6dce35e4d0da594f147a7f9bfc4 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:07.514533 4651 tablet_copy_service.cc:342] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Request end of tablet copy session db0742ee3860468590eefef6ae6074fd-41d6e6dce35e4d0da594f147a7f9bfc4 received from {username='slave'} at 127.29.219.70:45393
I20240501 20:10:07.515412 4651 tablet_copy_service.cc:433] P 1d34e18a1d394d3f91c9f50d1c7bea4e: ending tablet copy session db0742ee3860468590eefef6ae6074fd-41d6e6dce35e4d0da594f147a7f9bfc4 on tablet 41d6e6dce35e4d0da594f147a7f9bfc4 with peer db0742ee3860468590eefef6ae6074fd
I20240501 20:10:07.519685 4234 raft_consensus.cc:1261] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:07.520876 4365 raft_consensus.cc:1261] T ca4f195a6dd14be981395538c2a33ad9 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:07.520889 8153 consensus_queue.cc:1035] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:10:07.521209 4435 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.68:45205, user_credentials={real_user=slave}} blocked reactor thread for 78625us
I20240501 20:10:07.521898 4497 consensus_queue.cc:237] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:07.521993 8153 consensus_queue.cc:1035] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:07.524152 8492 consensus_queue.cc:1035] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:07.524904 7153 raft_consensus.cc:1261] T 9d165c29dabe4ae3be11ed84702c3cb2 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:07.525691 8153 consensus_queue.cc:1035] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:07.526186 4364 raft_consensus.cc:1261] T 9d165c29dabe4ae3be11ed84702c3cb2 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:07.526881 8153 consensus_queue.cc:1035] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:07.529516 8443 raft_consensus.cc:2932] T 9d165c29dabe4ae3be11ed84702c3cb2 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:07.531421 7153 raft_consensus.cc:2932] T 9d165c29dabe4ae3be11ed84702c3cb2 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:07.535121 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9d165c29dabe4ae3be11ed84702c3cb2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:07.536523 4364 raft_consensus.cc:2932] T 9d165c29dabe4ae3be11ed84702c3cb2 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:07.541332 4084 catalog_manager.cc:5484] T 9d165c29dabe4ae3be11ed84702c3cb2 P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 5 to 6, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:07.550797 8251 raft_consensus.cc:2932] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:07.553205 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d165c29dabe4ae3be11ed84702c3cb2 with delete_type TABLET_DATA_TOMBSTONED (TS 1d34e18a1d394d3f91c9f50d1c7bea4e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51236
W20240501 20:10:07.557267 4433 consensus_peers.cc:485] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b -> Peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Couldn't send request to peer bceeae1c79964acabdae8ff771bf0c97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca4f195a6dd14be981395538c2a33ad9. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:07.558737 4083 catalog_manager.cc:5484] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:07.561201 4365 raft_consensus.cc:2932] T ca4f195a6dd14be981395538c2a33ad9 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:07.566767 8503 tablet_replica.cc:330] T 9d165c29dabe4ae3be11ed84702c3cb2 P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:07.567096 8503 raft_consensus.cc:2229] T 9d165c29dabe4ae3be11ed84702c3cb2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:07.567782 8503 raft_consensus.cc:2258] T 9d165c29dabe4ae3be11ed84702c3cb2 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:07.567524 4234 raft_consensus.cc:2932] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:07.569304 8503 ts_tablet_manager.cc:1875] T 9d165c29dabe4ae3be11ed84702c3cb2 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:07.573438 7420 raft_consensus.cc:1203] T ef7c0a48f11945d3953f3461d06bc184 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:07.583225 8503 ts_tablet_manager.cc:1888] T 9d165c29dabe4ae3be11ed84702c3cb2 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:07.583500 8503 log.cc:1196] T 9d165c29dabe4ae3be11ed84702c3cb2 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/9d165c29dabe4ae3be11ed84702c3cb2
I20240501 20:10:07.584506 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet 9d165c29dabe4ae3be11ed84702c3cb2 (table rebalance_test_table_13 [id=e21b0047cbf842c6833a78db81a34171]) successfully deleted
I20240501 20:10:07.602299 4499 consensus_queue.cc:237] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:07.609290 4087 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:07.612818 4627 raft_consensus.cc:1261] T 2aa1a5f266e840698f2127289628dcba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:07.612982 4365 raft_consensus.cc:1261] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:07.614025 8443 consensus_queue.cc:1035] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:07.614651 8443 consensus_queue.cc:1035] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:10:07.615386 4435 consensus_peers.cc:485] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b -> Peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): Couldn't send request to peer b9957d4086bc4a76b18ed9265a5739d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2aa1a5f266e840698f2127289628dcba. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:07.622088 8479 tablet.cc:1722] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:07.627264 8483 tablet.cc:1722] T 38d1f8cee44d4a3a9c378733e3b146cf P 1d34e18a1d394d3f91c9f50d1c7bea4e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:07.645097 8486 tablet.cc:1722] T 38d1f8cee44d4a3a9c378733e3b146cf P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:07.646425 4083 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table_new_name [id=2fa59113bdc54d6aabf88f36456b3840] alter complete (version 3)
I20240501 20:10:07.674042 8251 raft_consensus.cc:2932] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:07.675016 4627 raft_consensus.cc:2932] T 2aa1a5f266e840698f2127289628dcba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:07.681867 4087 catalog_manager.cc:5484] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:07.685040 4364 raft_consensus.cc:2932] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:07.687347 4087 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:07.697678 4478 tablet_service.cc:1467] Processing CreateTablet for tablet 44aa74088ba24c338820957983d83e04 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=514d975194074c92815ac14399279d97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:07.698199 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44aa74088ba24c338820957983d83e04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:07.699313 4347 tablet_service.cc:1467] Processing CreateTablet for tablet 44aa74088ba24c338820957983d83e04 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=514d975194074c92815ac14399279d97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:07.699756 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44aa74088ba24c338820957983d83e04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:07.700371 7133 tablet_service.cc:1467] Processing CreateTablet for tablet 1d860bb204134f0994462275519feecf (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=514d975194074c92815ac14399279d97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:07.700834 7133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d860bb204134f0994462275519feecf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:07.701036 7134 tablet_service.cc:1467] Processing CreateTablet for tablet c147c98b674e499992f0b9f556ac6815 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=514d975194074c92815ac14399279d97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:07.701462 7134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c147c98b674e499992f0b9f556ac6815. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:07.703542 7266 tablet_service.cc:1467] Processing CreateTablet for tablet c147c98b674e499992f0b9f556ac6815 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=514d975194074c92815ac14399279d97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:07.704041 7266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c147c98b674e499992f0b9f556ac6815. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:07.704350 7400 tablet_service.cc:1467] Processing CreateTablet for tablet 44aa74088ba24c338820957983d83e04 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=514d975194074c92815ac14399279d97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:07.704806 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44aa74088ba24c338820957983d83e04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:07.708688 7000 tablet_service.cc:1467] Processing CreateTablet for tablet c147c98b674e499992f0b9f556ac6815 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=514d975194074c92815ac14399279d97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:07.709278 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c147c98b674e499992f0b9f556ac6815. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:07.709421 4477 tablet_service.cc:1467] Processing CreateTablet for tablet 1d860bb204134f0994462275519feecf (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=514d975194074c92815ac14399279d97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:07.709833 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d860bb204134f0994462275519feecf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:07.708693 7001 tablet_service.cc:1467] Processing CreateTablet for tablet 1d860bb204134f0994462275519feecf (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=514d975194074c92815ac14399279d97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:07.711383 7001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d860bb204134f0994462275519feecf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:07.725198 8381 ts_tablet_manager.cc:902] T 1c6c89fa7e6f485e889a5197da4bd135 P ef0a4ada548446b3a531cb39a3c4e38d: Initiating tablet copy from peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:07.739537 8516 tablet_bootstrap.cc:492] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:07.742039 8516 tablet_bootstrap.cc:654] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:07.746501 7284 raft_consensus.cc:1203] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:07.753158 4092 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:07.753432 4092 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:07.753445 8515 tablet_bootstrap.cc:492] T 44aa74088ba24c338820957983d83e04 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:07.757565 8515 tablet_bootstrap.cc:654] T 44aa74088ba24c338820957983d83e04 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:07.760105 4092 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 38d1f8cee44d4a3a9c378733e3b146cf
I20240501 20:10:07.760922 8314 raft_consensus.cc:1050] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e: attempting to promote NON_VOTER db0742ee3860468590eefef6ae6074fd to VOTER
I20240501 20:10:07.761410 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet 38d1f8cee44d4a3a9c378733e3b146cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:07 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:07.761752 8503 tablet_replica.cc:330] T 38d1f8cee44d4a3a9c378733e3b146cf P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:07.761974 8503 raft_consensus.cc:2229] T 38d1f8cee44d4a3a9c378733e3b146cf P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:07.762182 8314 consensus_queue.cc:237] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:07.762204 8503 raft_consensus.cc:2258] T 38d1f8cee44d4a3a9c378733e3b146cf P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:07.764577 8503 ts_tablet_manager.cc:1875] T 38d1f8cee44d4a3a9c378733e3b146cf P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:07.765753 7267 tablet_service.cc:1514] Processing DeleteTablet for tablet 38d1f8cee44d4a3a9c378733e3b146cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:07 UTC) from {username='slave'} at 127.0.0.1:54166
I20240501 20:10:07.765796 7400 tablet_service.cc:1514] Processing DeleteTablet for tablet 38d1f8cee44d4a3a9c378733e3b146cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:07 UTC) from {username='slave'} at 127.0.0.1:40182
I20240501 20:10:07.773052 8503 ts_tablet_manager.cc:1888] T 38d1f8cee44d4a3a9c378733e3b146cf P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:07.775095 8503 log.cc:1196] T 38d1f8cee44d4a3a9c378733e3b146cf P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/38d1f8cee44d4a3a9c378733e3b146cf
I20240501 20:10:07.776269 8503 ts_tablet_manager.cc:1909] T 38d1f8cee44d4a3a9c378733e3b146cf P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:07.778782 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet 38d1f8cee44d4a3a9c378733e3b146cf (table rebalancer_dynamic_table_new_name [id=2fa59113bdc54d6aabf88f36456b3840]) successfully deleted
I20240501 20:10:07.780313 8516 tablet_bootstrap.cc:492] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:07.780579 8516 ts_tablet_manager.cc:1367] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.041s user 0.000s sys 0.005s
I20240501 20:10:07.781764 8516 raft_consensus.cc:348] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:07.782485 8516 raft_consensus.cc:374] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:07.783087 8516 raft_consensus.cc:729] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:07.787623 7153 raft_consensus.cc:1261] T 7725fa9f98e9439797a532ed6f2f7dba P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:07.787806 4234 raft_consensus.cc:1261] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:07.788471 8071 consensus_queue.cc:1035] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:07.789506 8374 consensus_queue.cc:1035] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:07.790426 8514 tablet_bootstrap.cc:492] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:07.796861 4499 raft_consensus.cc:1261] T 7725fa9f98e9439797a532ed6f2f7dba P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:07.797362 8516 consensus_queue.cc:260] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:07.797647 8314 consensus_queue.cc:1035] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:07.799275 8514 tablet_bootstrap.cc:654] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:07.807134 8516 ts_tablet_manager.cc:1398] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.026s user 0.000s sys 0.002s
I20240501 20:10:07.864311 4087 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:07.866535 8366 raft_consensus.cc:1050] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd: attempting to promote NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e to VOTER
I20240501 20:10:07.867651 8366 consensus_queue.cc:237] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } }
I20240501 20:10:07.872427 7153 raft_consensus.cc:1203] T 41d6e6dce35e4d0da594f147a7f9bfc4 P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:07.876255 8153 raft_consensus.cc:1050] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b: attempting to promote NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d to VOTER
I20240501 20:10:07.877377 4215 tablet_service.cc:1467] Processing CreateTablet for tablet f6d6dc97540941b299e24d1ef0265627 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f13692743be04ba2803ebfdbd059b075]), partition=
I20240501 20:10:07.877254 8153 consensus_queue.cc:237] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:07.877802 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d6dc97540941b299e24d1ef0265627. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:07.877796 7265 tablet_service.cc:1467] Processing CreateTablet for tablet f6d6dc97540941b299e24d1ef0265627 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f13692743be04ba2803ebfdbd059b075]), partition=
I20240501 20:10:07.878247 7265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d6dc97540941b299e24d1ef0265627. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:07.887836 7000 tablet_service.cc:1467] Processing CreateTablet for tablet f6d6dc97540941b299e24d1ef0265627 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f13692743be04ba2803ebfdbd059b075]), partition=
I20240501 20:10:07.888612 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d6dc97540941b299e24d1ef0265627. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:07.906777 4627 raft_consensus.cc:1261] T bb85d7825c564339bd554deeded45cec P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:07.907620 8139 consensus_queue.cc:1035] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:07.914326 7020 raft_consensus.cc:1261] T bb85d7825c564339bd554deeded45cec P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:07.917384 8139 consensus_queue.cc:1035] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:07.917665 8533 tablet_bootstrap.cc:492] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:07.919381 8314 raft_consensus.cc:1050] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e: attempting to promote NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d to VOTER
I20240501 20:10:07.919636 7284 raft_consensus.cc:1261] T bb85d7825c564339bd554deeded45cec P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:07.919984 8533 tablet_bootstrap.cc:654] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:07.920365 8314 consensus_queue.cc:237] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:07.920481 8139 consensus_queue.cc:1035] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:07.922314 8526 tablet_replica.cc:330] T 38d1f8cee44d4a3a9c378733e3b146cf P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:07.922550 8526 raft_consensus.cc:2229] T 38d1f8cee44d4a3a9c378733e3b146cf P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:07.922827 8526 raft_consensus.cc:2258] T 38d1f8cee44d4a3a9c378733e3b146cf P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:07.924791 8526 ts_tablet_manager.cc:1875] T 38d1f8cee44d4a3a9c378733e3b146cf P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:07.927512 8525 tablet_replica.cc:330] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5: stopping tablet replica
I20240501 20:10:07.927920 8525 raft_consensus.cc:2229] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:07.928445 8525 raft_consensus.cc:2258] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:07.929670 8525 ts_tablet_manager.cc:1875] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:07.933712 8535 tablet_bootstrap.cc:492] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:07.938133 8430 raft_consensus.cc:482] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:07.938367 8430 raft_consensus.cc:504] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:07.939324 8538 tablet_bootstrap.cc:492] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:07.939843 8430 leader_election.cc:288] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:07.941162 4364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44aa74088ba24c338820957983d83e04" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
W20240501 20:10:07.941964 7355 leader_election.cc:341] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:07.942413 4497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44aa74088ba24c338820957983d83e04" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
W20240501 20:10:07.945103 7355 leader_election.cc:341] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:07.945257 7355 leader_election.cc:302] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5; no voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b
I20240501 20:10:07.945915 8430 raft_consensus.cc:2726] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:07.948397 8535 tablet_bootstrap.cc:654] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:07.955035 8374 raft_consensus.cc:2932] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:07.956121 4497 raft_consensus.cc:2932] T 7725fa9f98e9439797a532ed6f2f7dba P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:07.962148 8538 tablet_bootstrap.cc:654] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:07.965214 7153 raft_consensus.cc:2932] T 7725fa9f98e9439797a532ed6f2f7dba P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:07.966233 4087 catalog_manager.cc:5484] T 7725fa9f98e9439797a532ed6f2f7dba P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:07.968554 7335 raft_consensus.cc:1050] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a: attempting to promote NON_VOTER bceeae1c79964acabdae8ff771bf0c97 to VOTER
I20240501 20:10:07.969588 7335 consensus_queue.cc:237] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:07.975441 7287 raft_consensus.cc:1261] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:07.979091 4630 raft_consensus.cc:1261] T c66514deec5744da832bc030a582c46a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:07.983211 8305 consensus_queue.cc:1035] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:07.994773 7420 raft_consensus.cc:1203] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:07.998433 7022 raft_consensus.cc:1261] T 644d4d293e8c462297c6419965bd2c89 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:07.999542 8251 consensus_queue.cc:1035] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:08.007428 7335 consensus_queue.cc:1035] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:08.012737 8515 tablet_bootstrap.cc:492] T 44aa74088ba24c338820957983d83e04 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:10:08.013203 8515 ts_tablet_manager.cc:1367] T 44aa74088ba24c338820957983d83e04 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.267s user 0.021s sys 0.046s
I20240501 20:10:08.013870 4499 raft_consensus.cc:1261] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:08.019006 8552 raft_consensus.cc:2932] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:08.019104 8366 raft_consensus.cc:2932] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:08.019704 8305 consensus_queue.cc:1035] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:08.018376 8515 raft_consensus.cc:348] T 44aa74088ba24c338820957983d83e04 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:08.019989 8515 raft_consensus.cc:374] T 44aa74088ba24c338820957983d83e04 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:08.020128 8515 raft_consensus.cc:729] T 44aa74088ba24c338820957983d83e04 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:10:08.020419 8515 consensus_queue.cc:260] T 44aa74088ba24c338820957983d83e04 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:08.022023 7287 raft_consensus.cc:2932] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:08.023330 4630 raft_consensus.cc:2932] T bb85d7825c564339bd554deeded45cec P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:08.025753 8515 ts_tablet_manager.cc:1398] T 44aa74088ba24c338820957983d83e04 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240501 20:10:08.028589 7285 raft_consensus.cc:2932] T bb85d7825c564339bd554deeded45cec P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:08.031832 4364 raft_consensus.cc:1261] T 644d4d293e8c462297c6419965bd2c89 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:08.032260 4083 catalog_manager.cc:5484] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:08.032771 8251 consensus_queue.cc:1035] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:08.033051 8515 tablet_bootstrap.cc:492] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:08.035295 8515 tablet_bootstrap.cc:654] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:08.040663 4087 catalog_manager.cc:5484] T bb85d7825c564339bd554deeded45cec P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:08.048682 7021 raft_consensus.cc:1261] T fcff4be357084bf0903cfb1c0f62a866 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:08.049520 8071 consensus_queue.cc:1035] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:08.053385 4234 raft_consensus.cc:1261] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:08.060346 4499 raft_consensus.cc:1261] T fcff4be357084bf0903cfb1c0f62a866 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:08.061276 8314 consensus_queue.cc:1035] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:08.062477 8558 ts_tablet_manager.cc:902] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97: Initiating tablet copy from peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
W20240501 20:10:08.063843 4169 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.66:40689, user_credentials={real_user=slave}} blocked reactor thread for 114158us
I20240501 20:10:08.070299 8492 consensus_queue.cc:1035] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:08.080600 4627 consensus_queue.cc:237] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:08.083546 7153 raft_consensus.cc:1261] T 7725fa9f98e9439797a532ed6f2f7dba P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:08.083628 4235 raft_consensus.cc:1261] T fcff4be357084bf0903cfb1c0f62a866 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:08.084332 8374 consensus_queue.cc:1035] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:08.084339 4235 raft_consensus.cc:2932] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:08.088364 8374 raft_consensus.cc:2932] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:08.090945 8071 consensus_queue.cc:1035] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:08.094046 4079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7725fa9f98e9439797a532ed6f2f7dba with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:08.095137 8443 raft_consensus.cc:2932] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:08.095959 7153 raft_consensus.cc:2932] T 7725fa9f98e9439797a532ed6f2f7dba P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:08.096417 7022 raft_consensus.cc:2932] T 644d4d293e8c462297c6419965bd2c89 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:08.101845 4084 catalog_manager.cc:5484] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:08.105110 4364 raft_consensus.cc:2932] T 644d4d293e8c462297c6419965bd2c89 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:08.107725 7019 raft_consensus.cc:2932] T bb85d7825c564339bd554deeded45cec P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:08.109205 8515 tablet_bootstrap.cc:492] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:10:08.109382 8515 ts_tablet_manager.cc:1367] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.076s user 0.007s sys 0.008s
I20240501 20:10:08.110991 4497 raft_consensus.cc:2932] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:08.117039 8515 raft_consensus.cc:348] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:08.117416 8515 raft_consensus.cc:374] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:08.117518 8515 raft_consensus.cc:729] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:10:08.117825 8515 consensus_queue.cc:260] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:08.119437 8515 ts_tablet_manager.cc:1398] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.010s user 0.001s sys 0.002s
I20240501 20:10:08.125705 8538 tablet_bootstrap.cc:492] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:08.125921 8538 ts_tablet_manager.cc:1367] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.187s user 0.008s sys 0.008s
I20240501 20:10:08.127074 8538 raft_consensus.cc:348] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:08.127331 8538 raft_consensus.cc:374] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:08.127467 8538 raft_consensus.cc:729] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:08.127725 8538 consensus_queue.cc:260] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:08.128381 8538 ts_tablet_manager.cc:1398] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:08.132138 8514 tablet_bootstrap.cc:492] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:08.133366 8550 raft_consensus.cc:1050] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a: attempting to promote NON_VOTER bceeae1c79964acabdae8ff771bf0c97 to VOTER
I20240501 20:10:08.134518 8550 consensus_queue.cc:237] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:08.136260 4084 catalog_manager.cc:5484] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:08.138674 4234 raft_consensus.cc:1261] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:08.141355 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet 7725fa9f98e9439797a532ed6f2f7dba with delete_type TABLET_DATA_TOMBSTONED (TS 3e974c03d3c546f9ab6c91e762a5426b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:08.143659 8513 tablet_bootstrap.cc:492] T 44aa74088ba24c338820957983d83e04 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:08.151098 8558 tablet_copy_client.cc:308] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.67:34595
I20240501 20:10:08.154923 8374 consensus_queue.cc:1035] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:08.157400 4234 raft_consensus.cc:2932] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:08.163710 8514 ts_tablet_manager.cc:1367] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.373s user 0.015s sys 0.030s
I20240501 20:10:08.164665 8514 raft_consensus.cc:348] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:08.167671 8374 raft_consensus.cc:2932] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:08.174258 8533 tablet_bootstrap.cc:492] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:08.174605 8533 ts_tablet_manager.cc:1367] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.257s user 0.005s sys 0.000s
I20240501 20:10:08.175530 8533 raft_consensus.cc:348] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:08.177163 4498 raft_consensus.cc:2932] T fcff4be357084bf0903cfb1c0f62a866 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:08.180975 8535 tablet_bootstrap.cc:492] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:08.181218 8535 ts_tablet_manager.cc:1367] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.248s user 0.005s sys 0.000s
I20240501 20:10:08.181572 4519 tablet_copy_service.cc:140] P 3e974c03d3c546f9ab6c91e762a5426b: Received BeginTabletCopySession request for tablet ca4f195a6dd14be981395538c2a33ad9 from peer bceeae1c79964acabdae8ff771bf0c97 ({username='slave'} at 127.29.219.71:60861)
I20240501 20:10:08.181793 4519 tablet_copy_service.cc:161] P 3e974c03d3c546f9ab6c91e762a5426b: Beginning new tablet copy session on tablet ca4f195a6dd14be981395538c2a33ad9 from peer bceeae1c79964acabdae8ff771bf0c97 at {username='slave'} at 127.29.219.71:60861: session id = bceeae1c79964acabdae8ff771bf0c97-ca4f195a6dd14be981395538c2a33ad9
I20240501 20:10:08.186920 8514 raft_consensus.cc:374] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:08.187093 4519 tablet_copy_source_session.cc:205] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:08.187146 8514 raft_consensus.cc:729] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:08.187513 8514 consensus_queue.cc:260] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:08.157773 7153 consensus_queue.cc:237] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } }
I20240501 20:10:08.189491 4627 raft_consensus.cc:2932] T c66514deec5744da832bc030a582c46a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:08.189369 7021 raft_consensus.cc:2932] T fcff4be357084bf0903cfb1c0f62a866 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:08.190841 4087 catalog_manager.cc:5484] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:08.191745 8443 raft_consensus.cc:1050] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b: attempting to promote NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 to VOTER
I20240501 20:10:08.192538 8443 consensus_queue.cc:237] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:08.193534 8570 ts_tablet_manager.cc:902] T 2aa1a5f266e840698f2127289628dcba P b9957d4086bc4a76b18ed9265a5739d5: Initiating tablet copy from peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:08.196007 8513 tablet_bootstrap.cc:654] T 44aa74088ba24c338820957983d83e04 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:08.199146 4234 raft_consensus.cc:2932] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:08.201220 4630 consensus_queue.cc:237] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:08.188632 8558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4f195a6dd14be981395538c2a33ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:08.206053 4498 raft_consensus.cc:1261] T fcff4be357084bf0903cfb1c0f62a866 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:08.206701 8374 consensus_queue.cc:1035] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:08.209126 8071 raft_consensus.cc:2932] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:08.212990 8558 tablet_copy_client.cc:783] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:08.213328 8558 tablet_copy_client.cc:647] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:08.215272 8558 tablet_copy_client.cc:523] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:08.215862 7420 raft_consensus.cc:1261] T ef7c0a48f11945d3953f3461d06bc184 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:08.216399 8533 raft_consensus.cc:374] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:08.216480 8443 consensus_queue.cc:1035] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:08.216596 8533 raft_consensus.cc:729] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:08.217003 4364 raft_consensus.cc:1261] T ef7c0a48f11945d3953f3461d06bc184 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:08.217305 4627 raft_consensus.cc:1261] T ef7c0a48f11945d3953f3461d06bc184 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:08.217308 8533 consensus_queue.cc:260] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:08.217886 8492 consensus_queue.cc:1035] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:08.218113 4498 raft_consensus.cc:2932] T fcff4be357084bf0903cfb1c0f62a866 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:08.218446 8492 consensus_queue.cc:1035] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:08.219991 4079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fcff4be357084bf0903cfb1c0f62a866 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:08.226874 8381 tablet_copy_client.cc:308] T 1c6c89fa7e6f485e889a5197da4bd135 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.68:45205
I20240501 20:10:08.228339 8513 tablet_bootstrap.cc:492] T 44aa74088ba24c338820957983d83e04 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:10:08.228499 8513 ts_tablet_manager.cc:1367] T 44aa74088ba24c338820957983d83e04 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.086s user 0.007s sys 0.002s
I20240501 20:10:08.228480 4651 tablet_copy_service.cc:140] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Received BeginTabletCopySession request for tablet 1c6c89fa7e6f485e889a5197da4bd135 from peer ef0a4ada548446b3a531cb39a3c4e38d ({username='slave'} at 127.29.219.69:59565)
I20240501 20:10:08.228668 4651 tablet_copy_service.cc:161] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Beginning new tablet copy session on tablet 1c6c89fa7e6f485e889a5197da4bd135 from peer ef0a4ada548446b3a531cb39a3c4e38d at {username='slave'} at 127.29.219.69:59565: session id = ef0a4ada548446b3a531cb39a3c4e38d-1c6c89fa7e6f485e889a5197da4bd135
I20240501 20:10:08.229269 8513 raft_consensus.cc:348] T 44aa74088ba24c338820957983d83e04 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:08.229522 8513 raft_consensus.cc:374] T 44aa74088ba24c338820957983d83e04 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:08.229600 8513 raft_consensus.cc:729] T 44aa74088ba24c338820957983d83e04 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:10:08.229900 8513 consensus_queue.cc:260] T 44aa74088ba24c338820957983d83e04 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:08.230463 8513 ts_tablet_manager.cc:1398] T 44aa74088ba24c338820957983d83e04 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:08.231968 4651 tablet_copy_source_session.cc:205] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:08.233817 8381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6c89fa7e6f485e889a5197da4bd135. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:08.240937 4627 raft_consensus.cc:1261] T bb85d7825c564339bd554deeded45cec P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:08.241680 8366 consensus_queue.cc:1035] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:08.243945 4091 catalog_manager.cc:5484] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 5 to 6, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:08.244601 8366 raft_consensus.cc:2932] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:08.247687 4627 raft_consensus.cc:2932] T bb85d7825c564339bd554deeded45cec P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:08.256224 4076 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bb85d7825c564339bd554deeded45cec with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:08.261933 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet fcff4be357084bf0903cfb1c0f62a866 with delete_type TABLET_DATA_TOMBSTONED (TS dca57823003e4729b769ca12367d7530 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:08.264955 4087 catalog_manager.cc:5484] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:08.265938 8575 tablet_replica.cc:330] T 7725fa9f98e9439797a532ed6f2f7dba P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:08.261955 7019 raft_consensus.cc:1261] T bb85d7825c564339bd554deeded45cec P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:08.275175 8366 consensus_queue.cc:1035] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:08.281306 7019 raft_consensus.cc:2932] T bb85d7825c564339bd554deeded45cec P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:08.290409 4233 raft_consensus.cc:1261] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:08.290516 4499 raft_consensus.cc:1261] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:08.292178 8305 consensus_queue.cc:1035] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:08.292793 8305 consensus_queue.cc:1035] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:08.228271 8558 tablet_bootstrap.cc:492] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:08.291198 7284 raft_consensus.cc:1261] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:08.278342 7266 tablet_service.cc:1514] Processing DeleteTablet for tablet bb85d7825c564339bd554deeded45cec with delete_type TABLET_DATA_TOMBSTONED (TS bceeae1c79964acabdae8ff771bf0c97 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54166
I20240501 20:10:08.307825 8552 consensus_queue.cc:1035] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:08.219197 8533 ts_tablet_manager.cc:1398] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.044s user 0.002s sys 0.000s
I20240501 20:10:08.313371 8533 tablet_bootstrap.cc:492] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:08.315181 8533 tablet_bootstrap.cc:654] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:08.321524 8535 raft_consensus.cc:348] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:08.321907 8535 raft_consensus.cc:374] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:08.322088 8535 raft_consensus.cc:729] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:08.326001 8443 raft_consensus.cc:2932] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:08.327105 4631 raft_consensus.cc:2932] T ef7c0a48f11945d3953f3461d06bc184 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:08.328447 7421 raft_consensus.cc:2932] T ef7c0a48f11945d3953f3461d06bc184 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:08.333072 8588 tablet_replica.cc:330] T fcff4be357084bf0903cfb1c0f62a866 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:08.333357 8588 raft_consensus.cc:2229] T fcff4be357084bf0903cfb1c0f62a866 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:08.333559 8588 pending_rounds.cc:62] T fcff4be357084bf0903cfb1c0f62a866 P dca57823003e4729b769ca12367d7530: Trying to abort 1 pending ops.
I20240501 20:10:08.333655 8588 pending_rounds.cc:68] T fcff4be357084bf0903cfb1c0f62a866 P dca57823003e4729b769ca12367d7530: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022977875639164928 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "fcff4be357084bf0903cfb1c0f62a866" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } } }
I20240501 20:10:08.334173 8588 raft_consensus.cc:2894] T fcff4be357084bf0903cfb1c0f62a866 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 20:10:08.334308 8588 raft_consensus.cc:2866] T fcff4be357084bf0903cfb1c0f62a866 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:10:08.334434 8588 raft_consensus.cc:2258] T fcff4be357084bf0903cfb1c0f62a866 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:08.335704 4091 catalog_manager.cc:5484] T ef7c0a48f11945d3953f3461d06bc184 P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:08.340016 8575 raft_consensus.cc:2229] T 7725fa9f98e9439797a532ed6f2f7dba P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:08.339825 8535 consensus_queue.cc:260] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:08.340368 8575 raft_consensus.cc:2258] T 7725fa9f98e9439797a532ed6f2f7dba P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:08.341394 8575 ts_tablet_manager.cc:1875] T 7725fa9f98e9439797a532ed6f2f7dba P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:08.342607 8552 raft_consensus.cc:2932] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:08.345767 7022 raft_consensus.cc:1261] T fcff4be357084bf0903cfb1c0f62a866 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:08.347046 4364 raft_consensus.cc:2932] T ef7c0a48f11945d3953f3461d06bc184 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:08.347762 8588 ts_tablet_manager.cc:1875] T fcff4be357084bf0903cfb1c0f62a866 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:08.347846 8314 consensus_queue.cc:1035] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:08.351670 7022 raft_consensus.cc:2932] T fcff4be357084bf0903cfb1c0f62a866 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:08.354931 4497 raft_consensus.cc:2932] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:08.356321 8588 ts_tablet_manager.cc:1888] T fcff4be357084bf0903cfb1c0f62a866 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:08.356561 8588 log.cc:1196] T fcff4be357084bf0903cfb1c0f62a866 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/fcff4be357084bf0903cfb1c0f62a866
I20240501 20:10:08.357538 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet fcff4be357084bf0903cfb1c0f62a866 (table rebalance_test_table_1 [id=eccfbde58fa5455b9f89a20ad3a37f39]) successfully deleted
I20240501 20:10:08.357503 7284 raft_consensus.cc:2932] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:08.359992 4233 raft_consensus.cc:2932] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:08.372155 8525 ts_tablet_manager.cc:1888] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:08.372390 8525 log.cc:1196] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/wals/38d1f8cee44d4a3a9c378733e3b146cf
I20240501 20:10:08.373836 8525 ts_tablet_manager.cc:1909] T 38d1f8cee44d4a3a9c378733e3b146cf P b9957d4086bc4a76b18ed9265a5739d5: Deleting consensus metadata
I20240501 20:10:08.375180 8535 ts_tablet_manager.cc:1398] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.194s user 0.003s sys 0.000s
I20240501 20:10:08.375577 8535 tablet_bootstrap.cc:492] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:08.377440 8535 tablet_bootstrap.cc:654] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:08.386298 8575 ts_tablet_manager.cc:1888] T 7725fa9f98e9439797a532ed6f2f7dba P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:08.387025 8526 ts_tablet_manager.cc:1888] T 38d1f8cee44d4a3a9c378733e3b146cf P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:08.387252 8526 log.cc:1196] T 38d1f8cee44d4a3a9c378733e3b146cf P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/38d1f8cee44d4a3a9c378733e3b146cf
I20240501 20:10:08.387740 8575 log.cc:1196] T 7725fa9f98e9439797a532ed6f2f7dba P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/7725fa9f98e9439797a532ed6f2f7dba
I20240501 20:10:08.388793 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet 7725fa9f98e9439797a532ed6f2f7dba (table rebalance_test_table_1 [id=eccfbde58fa5455b9f89a20ad3a37f39]) successfully deleted
I20240501 20:10:08.390923 4084 catalog_manager.cc:5484] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:08.395078 8526 ts_tablet_manager.cc:1909] T 38d1f8cee44d4a3a9c378733e3b146cf P bceeae1c79964acabdae8ff771bf0c97: Deleting consensus metadata
I20240501 20:10:08.396440 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet 38d1f8cee44d4a3a9c378733e3b146cf (table rebalancer_dynamic_table_new_name [id=2fa59113bdc54d6aabf88f36456b3840]) successfully deleted
I20240501 20:10:08.398236 4497 consensus_queue.cc:237] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:08.400741 7420 raft_consensus.cc:1261] T ef7c0a48f11945d3953f3461d06bc184 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:08.401409 8492 consensus_queue.cc:1035] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:08.402456 4627 raft_consensus.cc:1261] T ef7c0a48f11945d3953f3461d06bc184 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:08.403064 8492 consensus_queue.cc:1035] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:08.403930 8552 raft_consensus.cc:482] T 44aa74088ba24c338820957983d83e04 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:08.404146 8552 raft_consensus.cc:504] T 44aa74088ba24c338820957983d83e04 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:08.404889 8552 leader_election.cc:288] T 44aa74088ba24c338820957983d83e04 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:08.405295 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44aa74088ba24c338820957983d83e04" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:08.405498 7419 raft_consensus.cc:2454] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:10:08.406297 4302 leader_election.cc:302] T 44aa74088ba24c338820957983d83e04 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, b9957d4086bc4a76b18ed9265a5739d5; no voters:
I20240501 20:10:08.407424 8443 raft_consensus.cc:2932] T ef7c0a48f11945d3953f3461d06bc184 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:08.407938 4079 catalog_manager.cc:4830] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): tablet 38d1f8cee44d4a3a9c378733e3b146cf (table rebalancer_dynamic_table_new_name [id=2fa59113bdc54d6aabf88f36456b3840]) successfully deleted
I20240501 20:10:08.410898 8552 raft_consensus.cc:2781] T 44aa74088ba24c338820957983d83e04 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:08.411031 8552 raft_consensus.cc:482] T 44aa74088ba24c338820957983d83e04 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:08.411181 8552 raft_consensus.cc:3037] T 44aa74088ba24c338820957983d83e04 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:08.413657 8526 tablet_replica.cc:330] T bb85d7825c564339bd554deeded45cec P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:08.417955 8570 tablet_copy_client.cc:308] T 2aa1a5f266e840698f2127289628dcba P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.67:34595
I20240501 20:10:08.419945 4627 raft_consensus.cc:2932] T ef7c0a48f11945d3953f3461d06bc184 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:08.420171 7420 raft_consensus.cc:2932] T ef7c0a48f11945d3953f3461d06bc184 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:08.423027 8552 raft_consensus.cc:504] T 44aa74088ba24c338820957983d83e04 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:08.423272 4519 tablet_copy_service.cc:140] P 3e974c03d3c546f9ab6c91e762a5426b: Received BeginTabletCopySession request for tablet 2aa1a5f266e840698f2127289628dcba from peer b9957d4086bc4a76b18ed9265a5739d5 ({username='slave'} at 127.29.219.72:58885)
I20240501 20:10:08.423418 4519 tablet_copy_service.cc:161] P 3e974c03d3c546f9ab6c91e762a5426b: Beginning new tablet copy session on tablet 2aa1a5f266e840698f2127289628dcba from peer b9957d4086bc4a76b18ed9265a5739d5 at {username='slave'} at 127.29.219.72:58885: session id = b9957d4086bc4a76b18ed9265a5739d5-2aa1a5f266e840698f2127289628dcba
I20240501 20:10:08.424161 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44aa74088ba24c338820957983d83e04" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:10:08.424386 4499 raft_consensus.cc:3037] T 44aa74088ba24c338820957983d83e04 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:08.426474 8552 leader_election.cc:288] T 44aa74088ba24c338820957983d83e04 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:08.426553 4519 tablet_copy_source_session.cc:205] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:08.426673 7421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44aa74088ba24c338820957983d83e04" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:08.426957 7421 raft_consensus.cc:3037] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:08.429509 8570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aa1a5f266e840698f2127289628dcba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:08.432837 4499 raft_consensus.cc:2454] T 44aa74088ba24c338820957983d83e04 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:10:08.433357 4301 leader_election.cc:302] T 44aa74088ba24c338820957983d83e04 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:10:08.433404 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ef7c0a48f11945d3953f3461d06bc184 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:08.433756 4497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44aa74088ba24c338820957983d83e04" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:08.434032 4497 raft_consensus.cc:2362] T 44aa74088ba24c338820957983d83e04 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1. Re-sending same reply.
I20240501 20:10:08.434377 7421 raft_consensus.cc:2454] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:10:08.435101 8437 raft_consensus.cc:482] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:08.435286 8437 raft_consensus.cc:504] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:08.436085 8437 leader_election.cc:288] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:08.436152 8514 ts_tablet_manager.cc:1398] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.121s user 0.001s sys 0.002s
I20240501 20:10:08.436986 8526 raft_consensus.cc:2229] T bb85d7825c564339bd554deeded45cec P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:08.437320 8526 raft_consensus.cc:2258] T bb85d7825c564339bd554deeded45cec P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:08.437748 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c147c98b674e499992f0b9f556ac6815" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
W20240501 20:10:08.438369 7222 leader_election.cc:341] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:08.441438 4091 catalog_manager.cc:5484] T ef7c0a48f11945d3953f3461d06bc184 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:08.445302 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet ef7c0a48f11945d3953f3461d06bc184 with delete_type TABLET_DATA_TOMBSTONED (TS 10b47782558c41ee91f5e164a372049a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:08.446434 4364 consensus_queue.cc:237] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:08.448287 8548 raft_consensus.cc:482] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:08.448431 8548 raft_consensus.cc:504] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:08.449083 8548 leader_election.cc:288] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:08.449427 7284 raft_consensus.cc:1261] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:08.450337 4497 raft_consensus.cc:1261] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:08.451149 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d6dc97540941b299e24d1ef0265627" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:08.451400 8550 consensus_queue.cc:1035] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:08.451741 8535 tablet_bootstrap.cc:492] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:08.451936 8535 ts_tablet_manager.cc:1367] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.001s
W20240501 20:10:08.452129 4168 leader_election.cc:341] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:08.452777 8535 raft_consensus.cc:348] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:08.454427 8381 tablet_copy_client.cc:783] T 1c6c89fa7e6f485e889a5197da4bd135 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:08.455251 8381 tablet_copy_client.cc:647] T 1c6c89fa7e6f485e889a5197da4bd135 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:08.454505 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d6dc97540941b299e24d1ef0265627" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:08.455310 7335 consensus_queue.cc:1035] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 20:10:08.457630 4169 leader_election.cc:341] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:08.457783 4169 leader_election.cc:302] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dca57823003e4729b769ca12367d7530; no voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:08.458118 8544 raft_consensus.cc:2726] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:08.454888 8535 raft_consensus.cc:374] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:08.458242 8535 raft_consensus.cc:729] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:08.459446 8550 raft_consensus.cc:2932] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:08.462936 8535 consensus_queue.cc:260] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:08.464154 8533 tablet_bootstrap.cc:492] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:08.464390 8533 ts_tablet_manager.cc:1367] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.151s user 0.001s sys 0.004s
I20240501 20:10:08.465332 8533 raft_consensus.cc:348] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:08.465739 8533 raft_consensus.cc:374] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:08.465916 8533 raft_consensus.cc:729] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:08.465776 4497 raft_consensus.cc:2932] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:08.466341 8533 consensus_queue.cc:260] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:08.468477 8381 tablet_copy_client.cc:523] T 1c6c89fa7e6f485e889a5197da4bd135 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:08.470674 8533 ts_tablet_manager.cc:1398] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:10:08.470773 7284 raft_consensus.cc:2932] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:08.478071 8535 ts_tablet_manager.cc:1398] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.026s user 0.000s sys 0.002s
I20240501 20:10:08.481819 4364 consensus_queue.cc:237] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:08.482556 8535 tablet_bootstrap.cc:492] T f6d6dc97540941b299e24d1ef0265627 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:08.485332 7287 raft_consensus.cc:1261] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:08.486028 7335 consensus_queue.cc:1035] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:08.489112 8437 raft_consensus.cc:482] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:08.489310 8437 raft_consensus.cc:504] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:08.489012 8305 raft_consensus.cc:2932] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:08.490016 8437 leader_election.cc:288] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:08.491447 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d6dc97540941b299e24d1ef0265627" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:08.491739 4234 raft_consensus.cc:2454] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:08.491385 7287 raft_consensus.cc:2932] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:08.435032 8552 raft_consensus.cc:2781] T 44aa74088ba24c338820957983d83e04 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:08.492488 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d6dc97540941b299e24d1ef0265627" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
W20240501 20:10:08.493153 7222 leader_election.cc:341] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:08.493229 8570 tablet_copy_client.cc:783] T 2aa1a5f266e840698f2127289628dcba P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:08.493379 7222 leader_election.cc:302] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, dca57823003e4729b769ca12367d7530; no voters: ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:08.493496 8570 tablet_copy_client.cc:647] T 2aa1a5f266e840698f2127289628dcba P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:08.493685 8381 tablet_bootstrap.cc:492] T 1c6c89fa7e6f485e889a5197da4bd135 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:08.493947 8437 raft_consensus.cc:2781] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:08.494042 8437 raft_consensus.cc:482] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:08.494138 8437 raft_consensus.cc:3037] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:08.496351 4091 catalog_manager.cc:5484] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 5 to 6, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:08.497048 8437 raft_consensus.cc:504] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:08.497763 8437 leader_election.cc:288] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:08.503481 8570 tablet_copy_client.cc:523] T 2aa1a5f266e840698f2127289628dcba P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:08.504302 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d6dc97540941b299e24d1ef0265627" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:10:08.504608 4234 raft_consensus.cc:3037] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:08.506908 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d6dc97540941b299e24d1ef0265627" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
W20240501 20:10:08.507678 7222 leader_election.cc:341] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:08.508684 4234 raft_consensus.cc:2454] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:08.509127 8570 tablet_bootstrap.cc:492] T 2aa1a5f266e840698f2127289628dcba P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:08.509172 7222 leader_election.cc:302] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, dca57823003e4729b769ca12367d7530; no voters: ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:08.509690 8437 raft_consensus.cc:2781] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:08.510409 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c147c98b674e499992f0b9f556ac6815" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:08.510682 8139 raft_consensus.cc:482] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:08.512040 8535 tablet_bootstrap.cc:654] T f6d6dc97540941b299e24d1ef0265627 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:08.514433 8526 ts_tablet_manager.cc:1875] T bb85d7825c564339bd554deeded45cec P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:08.517885 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet f40b062e103d4e54a913fa0f4111f623 with delete_type TABLET_DATA_TOMBSTONED (TS dca57823003e4729b769ca12367d7530 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:08.518262 8588 tablet_replica.cc:330] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:08.518463 8588 raft_consensus.cc:2229] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:08.518781 8588 raft_consensus.cc:2258] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:08.521158 8588 ts_tablet_manager.cc:1875] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:08.526113 8601 tablet_replica.cc:330] T ef7c0a48f11945d3953f3461d06bc184 P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:08.526396 8601 raft_consensus.cc:2229] T ef7c0a48f11945d3953f3461d06bc184 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:08.526732 8601 raft_consensus.cc:2258] T ef7c0a48f11945d3953f3461d06bc184 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:08.527910 8601 ts_tablet_manager.cc:1875] T ef7c0a48f11945d3953f3461d06bc184 P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:08.528573 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c66514deec5744da832bc030a582c46a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:08.529687 8552 raft_consensus.cc:686] T 44aa74088ba24c338820957983d83e04 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Becoming Leader. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Running, Role: LEADER
I20240501 20:10:08.530301 8437 raft_consensus.cc:686] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Becoming Leader. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Running, Role: LEADER
I20240501 20:10:08.530659 8552 consensus_queue.cc:237] T 44aa74088ba24c338820957983d83e04 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:08.530735 8437 consensus_queue.cc:237] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:08.532917 8535 tablet_bootstrap.cc:492] T f6d6dc97540941b299e24d1ef0265627 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:08.533167 8535 ts_tablet_manager.cc:1367] T f6d6dc97540941b299e24d1ef0265627 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.051s user 0.000s sys 0.006s
I20240501 20:10:08.534088 8535 raft_consensus.cc:348] T f6d6dc97540941b299e24d1ef0265627 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:08.540120 8548 raft_consensus.cc:1050] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530: attempting to promote NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 to VOTER
I20240501 20:10:08.540163 8535 raft_consensus.cc:374] T f6d6dc97540941b299e24d1ef0265627 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:08.540279 8535 raft_consensus.cc:729] T f6d6dc97540941b299e24d1ef0265627 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:08.541046 8535 consensus_queue.cc:260] T f6d6dc97540941b299e24d1ef0265627 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:08.541002 8548 consensus_queue.cc:237] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:08.543886 4497 raft_consensus.cc:1261] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:08.544780 8305 consensus_queue.cc:1035] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:08.547266 4084 catalog_manager.cc:5484] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71). New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:08.549091 8535 ts_tablet_manager.cc:1398] T f6d6dc97540941b299e24d1ef0265627 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20240501 20:10:08.552323 8526 ts_tablet_manager.cc:1888] T bb85d7825c564339bd554deeded45cec P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:08.552549 8526 log.cc:1196] T bb85d7825c564339bd554deeded45cec P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/bb85d7825c564339bd554deeded45cec
I20240501 20:10:08.555460 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet bb85d7825c564339bd554deeded45cec (table rebalancer_extra_table_0 [id=40f3268f46a14344bdf5de27989a6f1a]) successfully deleted
I20240501 20:10:08.557109 4497 raft_consensus.cc:2932] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:08.557590 7420 raft_consensus.cc:1261] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:08.558424 8544 consensus_queue.cc:1035] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:08.562674 8601 ts_tablet_manager.cc:1888] T ef7c0a48f11945d3953f3461d06bc184 P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:08.564239 8495 raft_consensus.cc:482] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:08.564461 8495 raft_consensus.cc:504] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:08.565557 8495 leader_election.cc:288] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:08.566921 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d860bb204134f0994462275519feecf" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:08.567227 4499 raft_consensus.cc:2454] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:08.567570 4364 raft_consensus.cc:1261] T d690199665174c7d9bdbd54d2af4ac70 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:08.568259 6957 leader_election.cc:302] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:08.568876 4499 raft_consensus.cc:1261] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:08.569051 8548 consensus_queue.cc:1035] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:08.569304 8495 raft_consensus.cc:2781] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:08.569425 8495 raft_consensus.cc:482] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:08.569512 8495 raft_consensus.cc:3037] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:08.573215 8495 raft_consensus.cc:504] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:08.574579 8546 consensus_queue.cc:1035] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:08.576295 8601 log.cc:1196] T ef7c0a48f11945d3953f3461d06bc184 P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/ef7c0a48f11945d3953f3461d06bc184
I20240501 20:10:08.577272 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet ef7c0a48f11945d3953f3461d06bc184 (table rebalance_test_table_9 [id=eef6790ef9874038b63d81f758f188d7]) successfully deleted
I20240501 20:10:08.577864 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d860bb204134f0994462275519feecf" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:10:08.578330 4498 raft_consensus.cc:3037] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:08.580279 8558 tablet_bootstrap.cc:492] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:08.580682 8558 tablet_bootstrap.cc:492] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap complete.
I20240501 20:10:08.580971 8558 ts_tablet_manager.cc:1367] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.353s user 0.054s sys 0.003s
I20240501 20:10:08.581840 8558 raft_consensus.cc:348] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:08.582080 8558 raft_consensus.cc:729] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: LEARNER
I20240501 20:10:08.582301 8558 consensus_queue.cc:260] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:08.588742 4498 raft_consensus.cc:2454] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:08.590562 8532 raft_consensus.cc:2932] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:08.591405 6957 leader_election.cc:302] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:08.591320 4364 raft_consensus.cc:2932] T d690199665174c7d9bdbd54d2af4ac70 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:08.593132 4499 raft_consensus.cc:2932] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:08.598160 8617 raft_consensus.cc:2781] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:08.599803 8495 leader_election.cc:288] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:08.603271 4087 catalog_manager.cc:5484] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:08.605059 7421 raft_consensus.cc:2932] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:08.614419 8617 raft_consensus.cc:686] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Becoming Leader. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Running, Role: LEADER
I20240501 20:10:08.614939 8617 consensus_queue.cc:237] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:08.628674 8314 raft_consensus.cc:1050] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e: attempting to promote NON_VOTER db0742ee3860468590eefef6ae6074fd to VOTER
I20240501 20:10:08.629765 8314 consensus_queue.cc:237] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:08.630153 8558 ts_tablet_manager.cc:1398] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.049s user 0.003s sys 0.001s
I20240501 20:10:08.631031 4519 tablet_copy_service.cc:342] P 3e974c03d3c546f9ab6c91e762a5426b: Request end of tablet copy session bceeae1c79964acabdae8ff771bf0c97-ca4f195a6dd14be981395538c2a33ad9 received from {username='slave'} at 127.29.219.71:60861
I20240501 20:10:08.631191 4519 tablet_copy_service.cc:433] P 3e974c03d3c546f9ab6c91e762a5426b: ending tablet copy session bceeae1c79964acabdae8ff771bf0c97-ca4f195a6dd14be981395538c2a33ad9 on tablet ca4f195a6dd14be981395538c2a33ad9 with peer bceeae1c79964acabdae8ff771bf0c97
I20240501 20:10:08.632759 8139 raft_consensus.cc:504] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:08.638087 8139 leader_election.cc:288] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:08.642741 7154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d860bb204134f0994462275519feecf" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
W20240501 20:10:08.643649 6955 leader_election.cc:341] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:08.648339 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c147c98b674e499992f0b9f556ac6815" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:08.648692 7020 raft_consensus.cc:2454] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:08.654345 7155 raft_consensus.cc:2454] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:08.654417 7090 leader_election.cc:302] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db0742ee3860468590eefef6ae6074fd, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:08.654927 8139 raft_consensus.cc:2781] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:08.655078 8139 raft_consensus.cc:482] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:08.655210 8139 raft_consensus.cc:3037] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:08.657824 8514 tablet_bootstrap.cc:492] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:08.659456 8139 raft_consensus.cc:504] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:08.661960 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c147c98b674e499992f0b9f556ac6815" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:08.662303 7020 raft_consensus.cc:3037] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:08.665066 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d860bb204134f0994462275519feecf" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:08.625178 4234 consensus_queue.cc:237] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:08.666596 8514 tablet_bootstrap.cc:654] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:08.666864 7020 raft_consensus.cc:2454] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
W20240501 20:10:08.667409 6955 leader_election.cc:341] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:08.667805 7090 leader_election.cc:302] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db0742ee3860468590eefef6ae6074fd, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:08.672199 8366 raft_consensus.cc:2781] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:08.672394 8366 raft_consensus.cc:686] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Becoming Leader. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Running, Role: LEADER
I20240501 20:10:08.672837 8366 consensus_queue.cc:237] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:08.678781 7220 leader_election.cc:302] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, db0742ee3860468590eefef6ae6074fd; no voters: ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:08.679435 4091 catalog_manager.cc:5484] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69). New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:08.679822 8607 raft_consensus.cc:2781] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:08.679915 8607 raft_consensus.cc:482] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:08.680011 8607 raft_consensus.cc:3037] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:08.680655 4083 catalog_manager.cc:5484] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd reported cstate change: term changed from 0 to 1, leader changed from <none> to db0742ee3860468590eefef6ae6074fd (127.29.219.70). New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:08.668093 8139 leader_election.cc:288] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Requested vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:08.683509 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c147c98b674e499992f0b9f556ac6815" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:08.687134 8607 raft_consensus.cc:504] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:08.687757 7284 raft_consensus.cc:2379] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate db0742ee3860468590eefef6ae6074fd in current term 1: Already voted for candidate bceeae1c79964acabdae8ff771bf0c97 in this term.
I20240501 20:10:08.691609 8607 leader_election.cc:288] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:08.693269 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c147c98b674e499992f0b9f556ac6815" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:08.693614 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c147c98b674e499992f0b9f556ac6815" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:08.694278 7020 raft_consensus.cc:2379] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bceeae1c79964acabdae8ff771bf0c97 in current term 1: Already voted for candidate db0742ee3860468590eefef6ae6074fd in this term.
I20240501 20:10:08.694911 7222 leader_election.cc:302] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97; no voters: db0742ee3860468590eefef6ae6074fd, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:08.695602 8607 raft_consensus.cc:2726] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 20:10:08.696015 4499 raft_consensus.cc:1261] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:08.695997 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c147c98b674e499992f0b9f556ac6815" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:08.696278 7287 raft_consensus.cc:2379] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db0742ee3860468590eefef6ae6074fd in current term 1: Already voted for candidate bceeae1c79964acabdae8ff771bf0c97 in this term.
I20240501 20:10:08.697186 7420 raft_consensus.cc:1261] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:08.699802 8532 consensus_queue.cc:1035] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:08.702250 8521 consensus_queue.cc:1035] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:08.703820 8553 raft_consensus.cc:2932] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:08.708438 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d690199665174c7d9bdbd54d2af4ac70 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:08.713356 7420 raft_consensus.cc:2932] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:08.724575 4091 catalog_manager.cc:5484] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index 5 to 6, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:08.731210 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet c66514deec5744da832bc030a582c46a with delete_type TABLET_DATA_TOMBSTONED (TS 1d34e18a1d394d3f91c9f50d1c7bea4e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:08.734905 4498 raft_consensus.cc:2932] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:08.736272 4084 catalog_manager.cc:5484] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:08.741325 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet d690199665174c7d9bdbd54d2af4ac70 with delete_type TABLET_DATA_TOMBSTONED (TS 10b47782558c41ee91f5e164a372049a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:08.742339 8601 tablet_replica.cc:330] T d690199665174c7d9bdbd54d2af4ac70 P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:08.742552 8601 raft_consensus.cc:2229] T d690199665174c7d9bdbd54d2af4ac70 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:08.742938 8601 raft_consensus.cc:2258] T d690199665174c7d9bdbd54d2af4ac70 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:08.744139 8601 ts_tablet_manager.cc:1875] T d690199665174c7d9bdbd54d2af4ac70 P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:08.749601 8625 tablet_replica.cc:330] T c66514deec5744da832bc030a582c46a P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:08.749830 8625 raft_consensus.cc:2229] T c66514deec5744da832bc030a582c46a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:08.750100 8625 raft_consensus.cc:2258] T c66514deec5744da832bc030a582c46a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:08.751626 8601 ts_tablet_manager.cc:1888] T d690199665174c7d9bdbd54d2af4ac70 P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:08.751873 8601 log.cc:1196] T d690199665174c7d9bdbd54d2af4ac70 P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/d690199665174c7d9bdbd54d2af4ac70
I20240501 20:10:08.752837 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet d690199665174c7d9bdbd54d2af4ac70 (table rebalance_test_table_3 [id=5fb9ba92237649f89c99e4487cd45f29]) successfully deleted
I20240501 20:10:08.761186 8514 tablet_bootstrap.cc:492] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:08.761488 8514 ts_tablet_manager.cc:1367] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.270s user 0.003s sys 0.002s
I20240501 20:10:08.762421 8514 raft_consensus.cc:348] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:08.762797 8514 raft_consensus.cc:374] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:08.762940 8514 raft_consensus.cc:729] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:08.763397 8514 consensus_queue.cc:260] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:08.767529 8588 ts_tablet_manager.cc:1888] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:08.767670 8588 log.cc:1196] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/f40b062e103d4e54a913fa0f4111f623
I20240501 20:10:08.768476 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet f40b062e103d4e54a913fa0f4111f623 (table rebalance_test_table_7 [id=59dd8401cbd048cea30daf86baac8cfa]) successfully deleted
I20240501 20:10:08.758553 8625 ts_tablet_manager.cc:1875] T c66514deec5744da832bc030a582c46a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:08.776717 8625 ts_tablet_manager.cc:1888] T c66514deec5744da832bc030a582c46a P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:08.777494 8625 log.cc:1196] T c66514deec5744da832bc030a582c46a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/c66514deec5744da832bc030a582c46a
I20240501 20:10:08.778452 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet c66514deec5744da832bc030a582c46a (table rebalance_test_table_11 [id=3b02a4c6c0f54681a87f5ca2381de01d]) successfully deleted
I20240501 20:10:08.778705 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f40b062e103d4e54a913fa0f4111f623 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:08.779773 8381 tablet_bootstrap.cc:492] T 1c6c89fa7e6f485e889a5197da4bd135 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:08.780336 8381 tablet_bootstrap.cc:492] T 1c6c89fa7e6f485e889a5197da4bd135 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap complete.
I20240501 20:10:08.780637 8381 ts_tablet_manager.cc:1367] T 1c6c89fa7e6f485e889a5197da4bd135 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.291s user 0.036s sys 0.016s
I20240501 20:10:08.780759 4364 raft_consensus.cc:1261] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:08.781303 7153 raft_consensus.cc:1261] T 41d6e6dce35e4d0da594f147a7f9bfc4 P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:08.781471 8381 raft_consensus.cc:348] T 1c6c89fa7e6f485e889a5197da4bd135 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:08.781765 8381 raft_consensus.cc:729] T 1c6c89fa7e6f485e889a5197da4bd135 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: LEARNER
I20240501 20:10:08.781518 4091 catalog_manager.cc:5484] T 44aa74088ba24c338820957983d83e04 P 10b47782558c41ee91f5e164a372049a reported cstate change: term changed from 0 to 1, leader changed from <none> to 10b47782558c41ee91f5e164a372049a (127.29.219.66). New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:08.782068 8381 consensus_queue.cc:260] T 1c6c89fa7e6f485e889a5197da4bd135 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:08.782248 8071 consensus_queue.cc:1035] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:08.782701 8381 ts_tablet_manager.cc:1398] T 1c6c89fa7e6f485e889a5197da4bd135 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:08.783020 8514 ts_tablet_manager.cc:1398] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20240501 20:10:08.783205 8374 consensus_queue.cc:1035] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:08.785676 4651 tablet_copy_service.cc:342] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Request end of tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-1c6c89fa7e6f485e889a5197da4bd135 received from {username='slave'} at 127.29.219.69:59565
I20240501 20:10:08.785790 4651 tablet_copy_service.cc:433] P 1d34e18a1d394d3f91c9f50d1c7bea4e: ending tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-1c6c89fa7e6f485e889a5197da4bd135 on tablet 1c6c89fa7e6f485e889a5197da4bd135 with peer ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:08.791044 4497 raft_consensus.cc:1261] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:08.791810 8071 consensus_queue.cc:1035] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:08.799969 8592 raft_consensus.cc:2932] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:08.800873 4364 raft_consensus.cc:2932] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:08.807737 4084 catalog_manager.cc:5484] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:08.813306 8570 tablet_bootstrap.cc:492] T 2aa1a5f266e840698f2127289628dcba P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:08.815369 8570 tablet_bootstrap.cc:492] T 2aa1a5f266e840698f2127289628dcba P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap complete.
I20240501 20:10:08.815750 8570 ts_tablet_manager.cc:1367] T 2aa1a5f266e840698f2127289628dcba P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.307s user 0.052s sys 0.005s
I20240501 20:10:08.815192 7153 raft_consensus.cc:2932] T 41d6e6dce35e4d0da594f147a7f9bfc4 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:08.816026 4627 consensus_queue.cc:237] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:08.817107 8570 raft_consensus.cc:348] T 2aa1a5f266e840698f2127289628dcba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:08.818378 4365 raft_consensus.cc:1261] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:08.819648 8071 consensus_queue.cc:1035] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:08.820619 8570 raft_consensus.cc:729] T 2aa1a5f266e840698f2127289628dcba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: LEARNER
I20240501 20:10:08.821381 8570 consensus_queue.cc:260] T 2aa1a5f266e840698f2127289628dcba P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:08.825341 8570 ts_tablet_manager.cc:1398] T 2aa1a5f266e840698f2127289628dcba P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 20:10:08.826437 4519 tablet_copy_service.cc:342] P 3e974c03d3c546f9ab6c91e762a5426b: Request end of tablet copy session b9957d4086bc4a76b18ed9265a5739d5-2aa1a5f266e840698f2127289628dcba received from {username='slave'} at 127.29.219.72:58885
I20240501 20:10:08.826591 4519 tablet_copy_service.cc:433] P 3e974c03d3c546f9ab6c91e762a5426b: ending tablet copy session b9957d4086bc4a76b18ed9265a5739d5-2aa1a5f266e840698f2127289628dcba on tablet 2aa1a5f266e840698f2127289628dcba with peer b9957d4086bc4a76b18ed9265a5739d5
I20240501 20:10:08.829084 4083 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:08.829236 4365 raft_consensus.cc:2932] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:08.830299 7155 raft_consensus.cc:1261] T 41d6e6dce35e4d0da594f147a7f9bfc4 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:08.834266 8374 raft_consensus.cc:2932] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:08.834836 8071 consensus_queue.cc:1035] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:08.840353 4079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 41d6e6dce35e4d0da594f147a7f9bfc4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:08.844831 4091 catalog_manager.cc:5484] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:08.851420 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet 41d6e6dce35e4d0da594f147a7f9bfc4 with delete_type TABLET_DATA_TOMBSTONED (TS 3e974c03d3c546f9ab6c91e762a5426b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:08.851962 8575 tablet_replica.cc:330] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:08.852254 8575 raft_consensus.cc:2229] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:08.852466 8575 pending_rounds.cc:62] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 3e974c03d3c546f9ab6c91e762a5426b: Trying to abort 1 pending ops.
I20240501 20:10:08.852615 8575 pending_rounds.cc:68] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 3e974c03d3c546f9ab6c91e762a5426b: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022977878544805888 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "41d6e6dce35e4d0da594f147a7f9bfc4" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } } }
I20240501 20:10:08.853245 8575 raft_consensus.cc:2894] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 20:10:08.853394 8575 raft_consensus.cc:2866] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:10:08.853552 8575 raft_consensus.cc:2258] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:08.857650 8575 ts_tablet_manager.cc:1875] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:08.866914 7020 raft_consensus.cc:3037] T f6d6dc97540941b299e24d1ef0265627 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:08.866937 4234 raft_consensus.cc:1261] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:08.869550 7020 raft_consensus.cc:1261] T f6d6dc97540941b299e24d1ef0265627 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:08.870229 8437 consensus_queue.cc:1035] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:08.870733 8607 consensus_queue.cc:1035] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:08.889678 4083 catalog_manager.cc:5576] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237) does not have the latest schema for tablet f6d6dc97540941b299e24d1ef0265627 (table rebalancer_dynamic_table [id=f13692743be04ba2803ebfdbd059b075]). Expected version 1 got 0
I20240501 20:10:08.885963 7153 raft_consensus.cc:2932] T 41d6e6dce35e4d0da594f147a7f9bfc4 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:08.896142 8575 ts_tablet_manager.cc:1888] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:08.896442 8575 log.cc:1196] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/41d6e6dce35e4d0da594f147a7f9bfc4
I20240501 20:10:08.899039 8640 tablet.cc:1722] T f6d6dc97540941b299e24d1ef0265627 P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:08.899892 8640 tablet.cc:1620] T f6d6dc97540941b299e24d1ef0265627 P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:08.897521 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet 41d6e6dce35e4d0da594f147a7f9bfc4 (table rebalance_test_table_5 [id=1e137c7380c64655a3b011211c536d72]) successfully deleted
I20240501 20:10:08.905452 8639 tablet.cc:1722] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:08.906595 8639 tablet.cc:1620] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:08.912499 4091 catalog_manager.cc:5576] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941) does not have the latest schema for tablet f6d6dc97540941b299e24d1ef0265627 (table rebalancer_dynamic_table [id=f13692743be04ba2803ebfdbd059b075]). Expected version 1 got 0
I20240501 20:10:08.913446 4084 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=f13692743be04ba2803ebfdbd059b075] alter complete (version 1)
I20240501 20:10:08.919325 8643 tablet.cc:1722] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:08.921110 8643 tablet.cc:1620] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530: MemRowSet was empty: no flush needed.
I20240501 20:10:08.922179 4091 catalog_manager.cc:5576] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941) does not have the latest schema for tablet f6d6dc97540941b299e24d1ef0265627 (table rebalancer_dynamic_table [id=f13692743be04ba2803ebfdbd059b075]). Expected version 1 got 0
I20240501 20:10:08.926224 7850 raft_consensus.cc:482] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:08.926481 7850 raft_consensus.cc:504] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:08.927536 7850 leader_election.cc:288] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:08.928583 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d860bb204134f0994462275519feecf" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:08.929710 4497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d860bb204134f0994462275519feecf" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:08.930022 4497 raft_consensus.cc:2379] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db0742ee3860468590eefef6ae6074fd in current term 1: Already voted for candidate ef0a4ada548446b3a531cb39a3c4e38d in this term.
I20240501 20:10:08.931586 7090 leader_election.cc:302] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db0742ee3860468590eefef6ae6074fd; no voters: 3e974c03d3c546f9ab6c91e762a5426b, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:08.932121 7850 raft_consensus.cc:3037] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:08.934535 7850 raft_consensus.cc:2726] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:08.968091 8603 consensus_queue.cc:1035] T 44aa74088ba24c338820957983d83e04 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:08.985355 8604 consensus_queue.cc:1035] T 44aa74088ba24c338820957983d83e04 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:09.009908 7287 raft_consensus.cc:1203] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:09.018256 4091 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:09.026728 8639 tablet.cc:1722] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:09.027225 8639 tablet.cc:1620] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:09.029050 8643 tablet.cc:1722] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:09.029529 8643 tablet.cc:1620] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530: MemRowSet was empty: no flush needed.
I20240501 20:10:09.029788 8640 tablet.cc:1722] T f6d6dc97540941b299e24d1ef0265627 P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:09.030234 8640 tablet.cc:1620] T f6d6dc97540941b299e24d1ef0265627 P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:09.039687 8617 consensus_queue.cc:1035] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:09.043509 4084 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=f13692743be04ba2803ebfdbd059b075] alter complete (version 2)
I20240501 20:10:09.046037 8617 consensus_queue.cc:1035] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:09.078944 7850 consensus_queue.cc:1035] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:09.084918 8366 consensus_queue.cc:1035] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:09.094094 7420 raft_consensus.cc:1203] T 2aa1a5f266e840698f2127289628dcba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:09.121474 4497 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "644d4d293e8c462297c6419965bd2c89"
dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53682
I20240501 20:10:09.121743 4497 raft_consensus.cc:595] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Received request to transfer leadership
I20240501 20:10:09.147055 4091 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:09.156723 8639 tablet.cc:1722] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:09.157441 8640 tablet.cc:1722] T f6d6dc97540941b299e24d1ef0265627 P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:09.158190 8643 tablet.cc:1722] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:09.172024 4091 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table_new_name [id=f13692743be04ba2803ebfdbd059b075] alter complete (version 3)
I20240501 20:10:09.174741 8251 raft_consensus.cc:979] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b: : Instructing follower dca57823003e4729b769ca12367d7530 to start an election
I20240501 20:10:09.174923 8251 raft_consensus.cc:1067] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Signalling peer dca57823003e4729b769ca12367d7530 to start an election
I20240501 20:10:09.176568 4233 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "644d4d293e8c462297c6419965bd2c89"
dest_uuid: "dca57823003e4729b769ca12367d7530"
from {username='slave'} at 127.29.219.67:51285
I20240501 20:10:09.176904 4233 raft_consensus.cc:482] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:10:09.177115 4233 raft_consensus.cc:3037] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:09.188362 4233 raft_consensus.cc:504] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:09.192898 4497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "644d4d293e8c462297c6419965bd2c89" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:10:09.193141 4497 raft_consensus.cc:3032] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:10:09.193272 4497 raft_consensus.cc:729] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:10:09.193648 4497 consensus_queue.cc:260] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:09.194242 4497 raft_consensus.cc:3037] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:09.197219 4364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "644d4d293e8c462297c6419965bd2c89" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:10:09.197469 4364 raft_consensus.cc:3037] T 644d4d293e8c462297c6419965bd2c89 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:09.199659 4497 raft_consensus.cc:2454] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 2.
I20240501 20:10:09.200297 4364 raft_consensus.cc:2454] T 644d4d293e8c462297c6419965bd2c89 P 10b47782558c41ee91f5e164a372049a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 2.
I20240501 20:10:09.200755 4169 leader_election.cc:302] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:09.200984 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "644d4d293e8c462297c6419965bd2c89" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:09.201239 7020 raft_consensus.cc:3037] T 644d4d293e8c462297c6419965bd2c89 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:09.203953 8548 raft_consensus.cc:2781] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:09.204144 8548 raft_consensus.cc:686] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [term 2 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:10:09.204514 8548 consensus_queue.cc:237] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:09.208923 7020 raft_consensus.cc:2454] T 644d4d293e8c462297c6419965bd2c89 P ef0a4ada548446b3a531cb39a3c4e38d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 2.
I20240501 20:10:09.209416 4233 leader_election.cc:288] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 2 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:09.211301 4084 catalog_manager.cc:5484] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 1 to 2, leader changed from 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 2 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:09.274341 4084 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:09.274580 4084 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:09.278296 4084 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet f6d6dc97540941b299e24d1ef0265627
I20240501 20:10:09.279137 7000 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d6dc97540941b299e24d1ef0265627 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:09 UTC) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:09.279642 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d6dc97540941b299e24d1ef0265627 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:09 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:09.280122 7266 tablet_service.cc:1514] Processing DeleteTablet for tablet f6d6dc97540941b299e24d1ef0265627 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:09 UTC) from {username='slave'} at 127.0.0.1:54166
I20240501 20:10:09.290908 8746 tablet_replica.cc:330] T f6d6dc97540941b299e24d1ef0265627 P ef0a4ada548446b3a531cb39a3c4e38d: stopping tablet replica
I20240501 20:10:09.291265 8746 raft_consensus.cc:2229] T f6d6dc97540941b299e24d1ef0265627 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:09.291541 8746 raft_consensus.cc:2258] T f6d6dc97540941b299e24d1ef0265627 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:09.292865 8746 ts_tablet_manager.cc:1875] T f6d6dc97540941b299e24d1ef0265627 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:09.300223 8746 ts_tablet_manager.cc:1888] T f6d6dc97540941b299e24d1ef0265627 P ef0a4ada548446b3a531cb39a3c4e38d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:09.301986 8746 log.cc:1196] T f6d6dc97540941b299e24d1ef0265627 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/wals/f6d6dc97540941b299e24d1ef0265627
I20240501 20:10:09.302779 8746 ts_tablet_manager.cc:1909] T f6d6dc97540941b299e24d1ef0265627 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting consensus metadata
I20240501 20:10:09.304281 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet f6d6dc97540941b299e24d1ef0265627 (table rebalancer_dynamic_table_new_name [id=f13692743be04ba2803ebfdbd059b075]) successfully deleted
I20240501 20:10:09.311077 8748 tablet_replica.cc:330] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:09.311961 8747 tablet_replica.cc:330] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:09.311617 8748 raft_consensus.cc:2229] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:09.312601 8748 raft_consensus.cc:2258] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:09.312889 8747 raft_consensus.cc:2229] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:09.313114 8747 raft_consensus.cc:2258] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:09.314208 8748 ts_tablet_manager.cc:1875] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:09.319943 8747 ts_tablet_manager.cc:1875] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:09.323144 8748 ts_tablet_manager.cc:1888] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:09.323359 8748 log.cc:1196] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/f6d6dc97540941b299e24d1ef0265627
I20240501 20:10:09.323891 8748 ts_tablet_manager.cc:1909] T f6d6dc97540941b299e24d1ef0265627 P bceeae1c79964acabdae8ff771bf0c97: Deleting consensus metadata
I20240501 20:10:09.327176 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet f6d6dc97540941b299e24d1ef0265627 (table rebalancer_dynamic_table_new_name [id=f13692743be04ba2803ebfdbd059b075]) successfully deleted
I20240501 20:10:09.329706 8747 ts_tablet_manager.cc:1888] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:09.330119 8747 log.cc:1196] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/f6d6dc97540941b299e24d1ef0265627
I20240501 20:10:09.330991 8747 ts_tablet_manager.cc:1909] T f6d6dc97540941b299e24d1ef0265627 P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:09.335944 7020 raft_consensus.cc:1203] T 1c6c89fa7e6f485e889a5197da4bd135 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:09.336501 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet f6d6dc97540941b299e24d1ef0265627 (table rebalancer_dynamic_table_new_name [id=f13692743be04ba2803ebfdbd059b075]) successfully deleted
I20240501 20:10:09.347841 7020 consensus_queue.cc:237] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
W20240501 20:10:09.350436 6958 consensus_peers.cc:485] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d -> Peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Couldn't send request to peer 1d34e18a1d394d3f91c9f50d1c7bea4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3809cc8ffe6b4a9e99ebbb9d26a65279. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:09.350750 4364 raft_consensus.cc:1261] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:09.350911 7287 raft_consensus.cc:1261] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:09.351262 8617 consensus_queue.cc:1035] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:09.351581 8495 consensus_queue.cc:1035] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:09.354370 8617 raft_consensus.cc:2932] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:09.354924 7287 raft_consensus.cc:2932] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:09.358273 4364 raft_consensus.cc:2932] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:09.364862 4090 catalog_manager.cc:5484] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:09.368045 4083 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:09.378259 7133 tablet_service.cc:1467] Processing CreateTablet for tablet 04a5fb2b808d466491144e64eb54d0fd (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=4b4b9eb576884393874a7d89667da324]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:09.378873 7133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a5fb2b808d466491144e64eb54d0fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:09.379209 7400 tablet_service.cc:1467] Processing CreateTablet for tablet 264d3fcd3fae4acd8d76306c0f32ed72 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=4b4b9eb576884393874a7d89667da324]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:09.379640 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 264d3fcd3fae4acd8d76306c0f32ed72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:09.379634 4347 tablet_service.cc:1467] Processing CreateTablet for tablet 04a5fb2b808d466491144e64eb54d0fd (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=4b4b9eb576884393874a7d89667da324]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:09.379851 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 264d3fcd3fae4acd8d76306c0f32ed72 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=4b4b9eb576884393874a7d89667da324]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:09.380055 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a5fb2b808d466491144e64eb54d0fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:09.380254 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 264d3fcd3fae4acd8d76306c0f32ed72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:09.383028 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 0e8f897c4d744c468b33b1326b0c366a (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=4b4b9eb576884393874a7d89667da324]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:09.383409 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e8f897c4d744c468b33b1326b0c366a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:09.385354 4091 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:09.385499 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 04a5fb2b808d466491144e64eb54d0fd (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=4b4b9eb576884393874a7d89667da324]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:09.385918 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a5fb2b808d466491144e64eb54d0fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:09.386734 7134 tablet_service.cc:1467] Processing CreateTablet for tablet 264d3fcd3fae4acd8d76306c0f32ed72 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=4b4b9eb576884393874a7d89667da324]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:09.387311 7134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 264d3fcd3fae4acd8d76306c0f32ed72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:09.389945 7135 tablet_service.cc:1467] Processing CreateTablet for tablet 0e8f897c4d744c468b33b1326b0c366a (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=4b4b9eb576884393874a7d89667da324]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:09.389962 7266 tablet_service.cc:1467] Processing CreateTablet for tablet 0e8f897c4d744c468b33b1326b0c366a (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=4b4b9eb576884393874a7d89667da324]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:09.390373 7135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e8f897c4d744c468b33b1326b0c366a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:09.390450 7266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e8f897c4d744c468b33b1326b0c366a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:09.404492 8763 tablet_bootstrap.cc:492] T 0e8f897c4d744c468b33b1326b0c366a P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:09.406603 8761 tablet_bootstrap.cc:492] T 04a5fb2b808d466491144e64eb54d0fd P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:09.407491 8764 tablet_bootstrap.cc:492] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:09.407833 8763 tablet_bootstrap.cc:654] T 0e8f897c4d744c468b33b1326b0c366a P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:09.410058 8764 tablet_bootstrap.cc:654] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:09.410879 8761 tablet_bootstrap.cc:654] T 04a5fb2b808d466491144e64eb54d0fd P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:09.417086 8762 tablet_bootstrap.cc:492] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:09.419291 8764 tablet_bootstrap.cc:492] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:09.419533 8764 ts_tablet_manager.cc:1367] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20240501 20:10:09.420418 8762 tablet_bootstrap.cc:654] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:09.420341 8764 raft_consensus.cc:348] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:09.420565 8764 raft_consensus.cc:374] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:09.420702 8764 raft_consensus.cc:729] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:09.421015 8764 consensus_queue.cc:260] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:09.421943 7135 tablet_service.cc:1467] Processing CreateTablet for tablet 657466e8f25d4cc69ea61c80ffe9d210 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=90b1b66130bb43bfa55a6b400ea31fd1]), partition=
I20240501 20:10:09.422029 7000 tablet_service.cc:1467] Processing CreateTablet for tablet 657466e8f25d4cc69ea61c80ffe9d210 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=90b1b66130bb43bfa55a6b400ea31fd1]), partition=
I20240501 20:10:09.422374 7135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 657466e8f25d4cc69ea61c80ffe9d210. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:09.422417 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 657466e8f25d4cc69ea61c80ffe9d210. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:09.423359 8761 tablet_bootstrap.cc:492] T 04a5fb2b808d466491144e64eb54d0fd P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:10:09.423568 8761 ts_tablet_manager.cc:1367] T 04a5fb2b808d466491144e64eb54d0fd P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.006s
I20240501 20:10:09.424705 8761 raft_consensus.cc:348] T 04a5fb2b808d466491144e64eb54d0fd P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:09.425019 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 657466e8f25d4cc69ea61c80ffe9d210 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=90b1b66130bb43bfa55a6b400ea31fd1]), partition=
I20240501 20:10:09.425411 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 657466e8f25d4cc69ea61c80ffe9d210. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:09.427004 8763 tablet_bootstrap.cc:492] T 0e8f897c4d744c468b33b1326b0c366a P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:09.427160 8763 ts_tablet_manager.cc:1367] T 0e8f897c4d744c468b33b1326b0c366a P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20240501 20:10:09.428443 8763 raft_consensus.cc:348] T 0e8f897c4d744c468b33b1326b0c366a P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:09.428628 8763 raft_consensus.cc:374] T 0e8f897c4d744c468b33b1326b0c366a P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:09.428755 8763 raft_consensus.cc:729] T 0e8f897c4d744c468b33b1326b0c366a P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:09.429034 8763 consensus_queue.cc:260] T 0e8f897c4d744c468b33b1326b0c366a P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:09.431401 8763 ts_tablet_manager.cc:1398] T 0e8f897c4d744c468b33b1326b0c366a P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:10:09.431557 8761 raft_consensus.cc:374] T 04a5fb2b808d466491144e64eb54d0fd P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:09.431914 8763 tablet_bootstrap.cc:492] T 264d3fcd3fae4acd8d76306c0f32ed72 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:09.432060 8761 raft_consensus.cc:729] T 04a5fb2b808d466491144e64eb54d0fd P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:10:09.432370 8761 consensus_queue.cc:260] T 04a5fb2b808d466491144e64eb54d0fd P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:09.434203 8763 tablet_bootstrap.cc:654] T 264d3fcd3fae4acd8d76306c0f32ed72 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:09.434891 8761 ts_tablet_manager.cc:1398] T 04a5fb2b808d466491144e64eb54d0fd P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240501 20:10:09.437215 8762 tablet_bootstrap.cc:492] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:09.437470 8762 ts_tablet_manager.cc:1367] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240501 20:10:09.438881 8762 raft_consensus.cc:348] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:09.439343 8762 raft_consensus.cc:374] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:09.439477 8762 raft_consensus.cc:729] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:09.439742 8762 consensus_queue.cc:260] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:09.441759 8762 ts_tablet_manager.cc:1398] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:10:09.444583 8771 tablet_bootstrap.cc:492] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:09.446863 8771 tablet_bootstrap.cc:654] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:09.449131 8763 tablet_bootstrap.cc:492] T 264d3fcd3fae4acd8d76306c0f32ed72 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:09.449370 8763 ts_tablet_manager.cc:1367] T 264d3fcd3fae4acd8d76306c0f32ed72 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240501 20:10:09.450042 8366 raft_consensus.cc:482] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:09.450192 8366 raft_consensus.cc:504] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:09.450855 8763 raft_consensus.cc:348] T 264d3fcd3fae4acd8d76306c0f32ed72 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:09.451129 8763 raft_consensus.cc:374] T 264d3fcd3fae4acd8d76306c0f32ed72 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:09.451252 8366 leader_election.cc:288] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:09.451256 8763 raft_consensus.cc:729] T 264d3fcd3fae4acd8d76306c0f32ed72 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:09.451529 8763 consensus_queue.cc:260] T 264d3fcd3fae4acd8d76306c0f32ed72 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:09.452116 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e8f897c4d744c468b33b1326b0c366a" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:09.452492 8763 ts_tablet_manager.cc:1398] T 264d3fcd3fae4acd8d76306c0f32ed72 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:09.452980 8763 tablet_bootstrap.cc:492] T 657466e8f25d4cc69ea61c80ffe9d210 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:09.453366 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e8f897c4d744c468b33b1326b0c366a" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:09.453555 7420 raft_consensus.cc:2454] T 0e8f897c4d744c468b33b1326b0c366a P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:09.454906 8763 tablet_bootstrap.cc:654] T 657466e8f25d4cc69ea61c80ffe9d210 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
W20240501 20:10:09.452822 7089 leader_election.cc:341] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:09.455967 7091 leader_election.cc:302] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters: bceeae1c79964acabdae8ff771bf0c97
I20240501 20:10:09.456672 8366 raft_consensus.cc:2781] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:09.456856 8366 raft_consensus.cc:482] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:09.457010 8366 raft_consensus.cc:3037] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:09.458384 8771 tablet_bootstrap.cc:492] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:09.458598 8771 ts_tablet_manager.cc:1367] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.005s
I20240501 20:10:09.459692 8771 raft_consensus.cc:348] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:09.460551 8366 raft_consensus.cc:504] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:09.461863 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e8f897c4d744c468b33b1326b0c366a" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:09.462122 7420 raft_consensus.cc:3037] T 0e8f897c4d744c468b33b1326b0c366a P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:09.463117 8763 tablet_bootstrap.cc:492] T 657466e8f25d4cc69ea61c80ffe9d210 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:09.463128 8773 tablet_bootstrap.cc:492] T 0e8f897c4d744c468b33b1326b0c366a P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:09.463227 8771 raft_consensus.cc:374] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:09.463347 8763 ts_tablet_manager.cc:1367] T 657466e8f25d4cc69ea61c80ffe9d210 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20240501 20:10:09.463519 8771 raft_consensus.cc:729] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:09.464135 8771 consensus_queue.cc:260] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:09.464680 8763 raft_consensus.cc:348] T 657466e8f25d4cc69ea61c80ffe9d210 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:09.464934 8763 raft_consensus.cc:374] T 657466e8f25d4cc69ea61c80ffe9d210 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:09.465075 8763 raft_consensus.cc:729] T 657466e8f25d4cc69ea61c80ffe9d210 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:09.465415 7420 raft_consensus.cc:2454] T 0e8f897c4d744c468b33b1326b0c366a P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:09.465579 8763 consensus_queue.cc:260] T 657466e8f25d4cc69ea61c80ffe9d210 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:09.465965 7091 leader_election.cc:302] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:09.466265 8763 ts_tablet_manager.cc:1398] T 657466e8f25d4cc69ea61c80ffe9d210 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:10:09.466254 8139 raft_consensus.cc:2781] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:09.466398 8139 raft_consensus.cc:686] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Becoming Leader. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Running, Role: LEADER
I20240501 20:10:09.466495 8771 ts_tablet_manager.cc:1398] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240501 20:10:09.466668 8139 consensus_queue.cc:237] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:09.467885 8773 tablet_bootstrap.cc:654] T 0e8f897c4d744c468b33b1326b0c366a P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:09.470316 4090 catalog_manager.cc:5484] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd reported cstate change: term changed from 0 to 1, leader changed from <none> to db0742ee3860468590eefef6ae6074fd (127.29.219.70). New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:09.474023 8366 leader_election.cc:288] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Requested vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:09.474707 8764 ts_tablet_manager.cc:1398] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.055s user 0.000s sys 0.002s
I20240501 20:10:09.475032 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e8f897c4d744c468b33b1326b0c366a" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:09.475162 8764 tablet_bootstrap.cc:492] T 264d3fcd3fae4acd8d76306c0f32ed72 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
W20240501 20:10:09.476214 7089 leader_election.cc:341] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:09.476838 8764 tablet_bootstrap.cc:654] T 264d3fcd3fae4acd8d76306c0f32ed72 P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:09.476676 4630 consensus_queue.cc:237] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:09.486711 8773 tablet_bootstrap.cc:492] T 0e8f897c4d744c468b33b1326b0c366a P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:09.487103 8773 ts_tablet_manager.cc:1367] T 0e8f897c4d744c468b33b1326b0c366a P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.009s
I20240501 20:10:09.487550 8764 tablet_bootstrap.cc:492] T 264d3fcd3fae4acd8d76306c0f32ed72 P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:09.487884 8764 ts_tablet_manager.cc:1367] T 264d3fcd3fae4acd8d76306c0f32ed72 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20240501 20:10:09.488428 8773 raft_consensus.cc:348] T 0e8f897c4d744c468b33b1326b0c366a P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:09.488852 8773 raft_consensus.cc:374] T 0e8f897c4d744c468b33b1326b0c366a P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:09.488950 8764 raft_consensus.cc:348] T 264d3fcd3fae4acd8d76306c0f32ed72 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:09.489275 8764 raft_consensus.cc:374] T 264d3fcd3fae4acd8d76306c0f32ed72 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:09.489403 8764 raft_consensus.cc:729] T 264d3fcd3fae4acd8d76306c0f32ed72 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:09.489905 8773 raft_consensus.cc:729] T 0e8f897c4d744c468b33b1326b0c366a P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:09.489743 8764 consensus_queue.cc:260] T 264d3fcd3fae4acd8d76306c0f32ed72 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:09.490362 8773 consensus_queue.cc:260] T 0e8f897c4d744c468b33b1326b0c366a P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
W20240501 20:10:09.491068 4564 consensus_peers.cc:485] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f2576acd7d34ec1ad27ab04a863df10. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:09.493053 7287 raft_consensus.cc:1261] T 6f2576acd7d34ec1ad27ab04a863df10 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:09.493975 8764 ts_tablet_manager.cc:1398] T 264d3fcd3fae4acd8d76306c0f32ed72 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:10:09.494294 8764 tablet_bootstrap.cc:492] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:09.494699 4364 raft_consensus.cc:1261] T 6f2576acd7d34ec1ad27ab04a863df10 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:09.496369 8769 tablet_bootstrap.cc:492] T 657466e8f25d4cc69ea61c80ffe9d210 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:09.498183 8314 consensus_queue.cc:1035] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:09.498732 8592 consensus_queue.cc:1035] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:09.498869 8769 tablet_bootstrap.cc:654] T 657466e8f25d4cc69ea61c80ffe9d210 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:09.500569 8773 ts_tablet_manager.cc:1398] T 0e8f897c4d744c468b33b1326b0c366a P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.013s user 0.001s sys 0.002s
I20240501 20:10:09.502009 8764 tablet_bootstrap.cc:654] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:09.506070 8769 tablet_bootstrap.cc:492] T 657466e8f25d4cc69ea61c80ffe9d210 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:09.506314 8769 ts_tablet_manager.cc:1367] T 657466e8f25d4cc69ea61c80ffe9d210 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20240501 20:10:09.507700 8769 raft_consensus.cc:348] T 657466e8f25d4cc69ea61c80ffe9d210 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:09.507982 8769 raft_consensus.cc:374] T 657466e8f25d4cc69ea61c80ffe9d210 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:09.509384 8764 tablet_bootstrap.cc:492] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:09.509136 8314 raft_consensus.cc:2932] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:09.510051 8764 ts_tablet_manager.cc:1367] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240501 20:10:09.511101 8764 raft_consensus.cc:348] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:09.512017 8764 raft_consensus.cc:374] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:09.512212 8764 raft_consensus.cc:729] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:09.512490 8764 consensus_queue.cc:260] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:09.513173 8764 ts_tablet_manager.cc:1398] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:09.513394 8769 raft_consensus.cc:729] T 657466e8f25d4cc69ea61c80ffe9d210 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:09.513259 7284 raft_consensus.cc:2932] T 6f2576acd7d34ec1ad27ab04a863df10 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:09.513821 8764 tablet_bootstrap.cc:492] T 04a5fb2b808d466491144e64eb54d0fd P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:09.515458 8769 consensus_queue.cc:260] T 657466e8f25d4cc69ea61c80ffe9d210 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:09.516110 8769 ts_tablet_manager.cc:1398] T 657466e8f25d4cc69ea61c80ffe9d210 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240501 20:10:09.516754 8764 tablet_bootstrap.cc:654] T 04a5fb2b808d466491144e64eb54d0fd P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:09.519089 8764 tablet_bootstrap.cc:492] T 04a5fb2b808d466491144e64eb54d0fd P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:09.519325 8764 ts_tablet_manager.cc:1367] T 04a5fb2b808d466491144e64eb54d0fd P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240501 20:10:09.520186 8764 raft_consensus.cc:348] T 04a5fb2b808d466491144e64eb54d0fd P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:09.520553 8764 raft_consensus.cc:374] T 04a5fb2b808d466491144e64eb54d0fd P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:09.520694 8764 raft_consensus.cc:729] T 04a5fb2b808d466491144e64eb54d0fd P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:09.521107 8764 consensus_queue.cc:260] T 04a5fb2b808d466491144e64eb54d0fd P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:09.522033 8764 ts_tablet_manager.cc:1398] T 04a5fb2b808d466491144e64eb54d0fd P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:09.522598 8532 raft_consensus.cc:482] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:09.522802 8532 raft_consensus.cc:504] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:09.523507 8532 leader_election.cc:288] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:09.523944 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "264d3fcd3fae4acd8d76306c0f32ed72" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:09.524158 7153 raft_consensus.cc:2454] T 264d3fcd3fae4acd8d76306c0f32ed72 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:09.524412 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "264d3fcd3fae4acd8d76306c0f32ed72" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:09.524787 4167 leader_election.cc:302] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db0742ee3860468590eefef6ae6074fd, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:09.524935 7420 raft_consensus.cc:2454] T 264d3fcd3fae4acd8d76306c0f32ed72 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:09.525384 8548 raft_consensus.cc:2781] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:09.525502 8548 raft_consensus.cc:482] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:09.525591 8548 raft_consensus.cc:3037] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:09.528697 8548 raft_consensus.cc:504] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:09.530440 8548 leader_election.cc:288] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:09.531100 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "264d3fcd3fae4acd8d76306c0f32ed72" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:09.531155 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "264d3fcd3fae4acd8d76306c0f32ed72" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:09.531421 7153 raft_consensus.cc:3037] T 264d3fcd3fae4acd8d76306c0f32ed72 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:09.531420 7420 raft_consensus.cc:3037] T 264d3fcd3fae4acd8d76306c0f32ed72 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:09.532132 4083 catalog_manager.cc:5484] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:09.534493 7420 raft_consensus.cc:2454] T 264d3fcd3fae4acd8d76306c0f32ed72 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:09.536749 4170 leader_election.cc:302] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:09.537192 8548 raft_consensus.cc:2781] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:09.537380 8548 raft_consensus.cc:686] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:10:09.537730 8548 consensus_queue.cc:237] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:09.540002 8592 raft_consensus.cc:482] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:09.540241 8592 raft_consensus.cc:504] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:09.541179 8592 leader_election.cc:288] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:09.542747 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a5fb2b808d466491144e64eb54d0fd" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:09.543061 7155 raft_consensus.cc:2454] T 04a5fb2b808d466491144e64eb54d0fd P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:10:09.544023 4562 leader_election.cc:302] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:09.544584 8592 raft_consensus.cc:2781] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:09.546522 8592 raft_consensus.cc:482] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:09.546787 8592 raft_consensus.cc:3037] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:09.565588 8366 raft_consensus.cc:482] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:09.565800 8366 raft_consensus.cc:504] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:09.566576 8366 leader_election.cc:288] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:09.567692 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "657466e8f25d4cc69ea61c80ffe9d210" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:09.567924 7020 raft_consensus.cc:2454] T 657466e8f25d4cc69ea61c80ffe9d210 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:09.569233 7090 leader_election.cc:302] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db0742ee3860468590eefef6ae6074fd, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:09.570122 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "657466e8f25d4cc69ea61c80ffe9d210" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:09.570415 7420 raft_consensus.cc:2454] T 657466e8f25d4cc69ea61c80ffe9d210 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:09.570541 8139 raft_consensus.cc:2781] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:09.570679 8139 raft_consensus.cc:482] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:09.570803 8139 raft_consensus.cc:3037] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:09.551970 8592 raft_consensus.cc:504] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:09.572743 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a5fb2b808d466491144e64eb54d0fd" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:09.572942 7155 raft_consensus.cc:3037] T 04a5fb2b808d466491144e64eb54d0fd P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:09.573132 7153 raft_consensus.cc:2454] T 264d3fcd3fae4acd8d76306c0f32ed72 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:09.573563 8139 raft_consensus.cc:504] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:09.575479 8139 leader_election.cc:288] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:09.574813 4364 raft_consensus.cc:2932] T 6f2576acd7d34ec1ad27ab04a863df10 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:09.576411 7155 raft_consensus.cc:2454] T 04a5fb2b808d466491144e64eb54d0fd P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:10:09.577138 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "657466e8f25d4cc69ea61c80ffe9d210" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:09.577375 7020 raft_consensus.cc:3037] T 657466e8f25d4cc69ea61c80ffe9d210 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:09.582439 7020 raft_consensus.cc:2454] T 657466e8f25d4cc69ea61c80ffe9d210 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:09.583181 7421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "657466e8f25d4cc69ea61c80ffe9d210" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:09.583410 7421 raft_consensus.cc:3037] T 657466e8f25d4cc69ea61c80ffe9d210 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:09.586266 7421 raft_consensus.cc:2454] T 657466e8f25d4cc69ea61c80ffe9d210 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:09.587595 7091 leader_election.cc:302] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:09.589159 8366 raft_consensus.cc:2781] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:09.589342 8366 raft_consensus.cc:686] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Becoming Leader. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Running, Role: LEADER
I20240501 20:10:09.589280 4562 leader_election.cc:302] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:09.589736 8314 raft_consensus.cc:2781] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:09.589637 8366 consensus_queue.cc:237] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:09.589890 8314 raft_consensus.cc:686] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Becoming Leader. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Running, Role: LEADER
I20240501 20:10:09.590358 8314 consensus_queue.cc:237] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:09.592566 8251 raft_consensus.cc:1050] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b: attempting to promote NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 to VOTER
I20240501 20:10:09.593401 8251 consensus_queue.cc:237] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
W20240501 20:10:09.597613 4169 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.66:40689, user_credentials={real_user=slave}} blocked reactor thread for 59225.2us
I20240501 20:10:09.599778 4084 catalog_manager.cc:5484] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68). New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } }
W20240501 20:10:09.602797 4564 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.67:34595, user_credentials={real_user=slave}} blocked reactor thread for 77174.4us
I20240501 20:10:09.603483 4364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a5fb2b808d466491144e64eb54d0fd" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:10:09.603740 4364 raft_consensus.cc:2454] T 04a5fb2b808d466491144e64eb54d0fd P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:10:09.610985 4091 catalog_manager.cc:5484] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd reported cstate change: term changed from 0 to 1, leader changed from <none> to db0742ee3860468590eefef6ae6074fd (127.29.219.70). New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:09.615355 8592 leader_election.cc:288] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Requested vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:09.627453 8798 raft_consensus.cc:1050] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b: attempting to promote NON_VOTER bceeae1c79964acabdae8ff771bf0c97 to VOTER
I20240501 20:10:09.633800 4364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a5fb2b808d466491144e64eb54d0fd" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:10:09.634047 4364 raft_consensus.cc:3037] T 04a5fb2b808d466491144e64eb54d0fd P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:09.636180 4497 raft_consensus.cc:1261] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:10:09.628466 8798 consensus_queue.cc:237] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:09.636883 8532 consensus_queue.cc:1035] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:09.638592 4364 raft_consensus.cc:2454] T 04a5fb2b808d466491144e64eb54d0fd P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:10:09.648193 4364 raft_consensus.cc:1261] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:09.648304 7287 raft_consensus.cc:1261] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:09.648989 8799 consensus_queue.cc:1035] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:09.649638 8799 consensus_queue.cc:1035] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:09.651710 7420 raft_consensus.cc:1261] T 2aa1a5f266e840698f2127289628dcba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:09.653803 4498 consensus_queue.cc:237] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:09.654271 7020 raft_consensus.cc:1261] T 644d4d293e8c462297c6419965bd2c89 P ef0a4ada548446b3a531cb39a3c4e38d [term 2 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:10:09.654973 8532 consensus_queue.cc:1035] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:09.659790 8800 consensus_queue.cc:1035] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:09.662564 4627 raft_consensus.cc:1261] T 2aa1a5f266e840698f2127289628dcba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:09.665323 8492 consensus_queue.cc:1035] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 20:10:09.679375 4435 consensus_peers.cc:485] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b -> Peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): Couldn't send request to peer b9957d4086bc4a76b18ed9265a5739d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffaf0a0105ea4e1f9ed9d90c221ebc8a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:09.679198 4091 catalog_manager.cc:5484] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:09.680262 4233 raft_consensus.cc:1261] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:09.680603 4365 raft_consensus.cc:1261] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:09.680935 8804 consensus_queue.cc:1035] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:09.684057 8798 consensus_queue.cc:1035] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:09.685343 4364 raft_consensus.cc:1261] T ca4f195a6dd14be981395538c2a33ad9 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:09.686488 8800 consensus_queue.cc:1035] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:09.687235 8799 raft_consensus.cc:2932] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:09.687785 4365 raft_consensus.cc:2932] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:09.688997 4631 raft_consensus.cc:1261] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:09.695128 8804 consensus_queue.cc:1035] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:09.695752 4091 catalog_manager.cc:5484] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
W20240501 20:10:09.701483 4301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.65:42941, user_credentials={real_user=slave}} blocked reactor thread for 99605.4us
I20240501 20:10:09.718153 7284 raft_consensus.cc:2932] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:09.718047 8800 raft_consensus.cc:2932] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:09.723425 4233 raft_consensus.cc:2932] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:09.727352 8799 raft_consensus.cc:2932] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:09.729660 4627 raft_consensus.cc:2932] T 2aa1a5f266e840698f2127289628dcba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:09.726418 4091 catalog_manager.cc:5484] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:09.733508 4365 raft_consensus.cc:2932] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:09.734757 8641 raft_consensus.cc:482] T 0e8f897c4d744c468b33b1326b0c366a P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:09.734922 8641 raft_consensus.cc:504] T 0e8f897c4d744c468b33b1326b0c366a P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:09.735649 8641 leader_election.cc:288] T 0e8f897c4d744c468b33b1326b0c366a P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:09.736178 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e8f897c4d744c468b33b1326b0c366a" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:09.738020 7421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e8f897c4d744c468b33b1326b0c366a" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:09.738267 7421 raft_consensus.cc:2379] T 0e8f897c4d744c468b33b1326b0c366a P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bceeae1c79964acabdae8ff771bf0c97 in current term 1: Already voted for candidate db0742ee3860468590eefef6ae6074fd in this term.
I20240501 20:10:09.738711 7223 leader_election.cc:302] T 0e8f897c4d744c468b33b1326b0c366a P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97; no voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd
I20240501 20:10:09.737824 7420 raft_consensus.cc:2932] T 2aa1a5f266e840698f2127289628dcba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:09.739447 8641 raft_consensus.cc:3037] T 0e8f897c4d744c468b33b1326b0c366a P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:09.742163 8641 raft_consensus.cc:2726] T 0e8f897c4d744c468b33b1326b0c366a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:09.744217 4091 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:09.751025 4364 raft_consensus.cc:2932] T ca4f195a6dd14be981395538c2a33ad9 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:09.755153 4090 catalog_manager.cc:5484] T 2aa1a5f266e840698f2127289628dcba P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:09.762712 4365 raft_consensus.cc:1261] T 644d4d293e8c462297c6419965bd2c89 P 10b47782558c41ee91f5e164a372049a [term 2 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:10:09.763268 8548 consensus_queue.cc:1035] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:09.777745 4497 consensus_queue.cc:237] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:09.777904 4499 consensus_queue.cc:237] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:09.780642 7420 raft_consensus.cc:1261] T 2aa1a5f266e840698f2127289628dcba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:09.780771 4365 raft_consensus.cc:1261] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:09.781977 8251 consensus_queue.cc:1035] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:09.783231 8804 consensus_queue.cc:1035] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:09.787220 8251 raft_consensus.cc:2932] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:09.787572 4368 raft_consensus.cc:2932] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:09.796171 7287 raft_consensus.cc:1261] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:09.798137 4233 raft_consensus.cc:1261] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:09.800544 8799 consensus_queue.cc:1035] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:09.800400 7420 raft_consensus.cc:2932] T 2aa1a5f266e840698f2127289628dcba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:09.802192 8799 consensus_queue.cc:1035] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:09.803030 8801 raft_consensus.cc:2932] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:09.803740 4233 raft_consensus.cc:2932] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:09.806212 4631 raft_consensus.cc:2932] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:09.809206 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ca4f195a6dd14be981395538c2a33ad9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:09.811686 4083 catalog_manager.cc:5484] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:09.813414 7421 raft_consensus.cc:1261] T 657466e8f25d4cc69ea61c80ffe9d210 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:09.815922 8366 consensus_queue.cc:1035] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:09.817080 7287 raft_consensus.cc:2932] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:09.826171 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet ca4f195a6dd14be981395538c2a33ad9 with delete_type TABLET_DATA_TOMBSTONED (TS 10b47782558c41ee91f5e164a372049a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:09.827245 7020 raft_consensus.cc:1261] T 657466e8f25d4cc69ea61c80ffe9d210 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:09.827900 8366 consensus_queue.cc:1035] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:09.836977 4084 catalog_manager.cc:5484] T 2aa1a5f266e840698f2127289628dcba P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: config changed from index 5 to 6, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:09.858466 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet 2aa1a5f266e840698f2127289628dcba with delete_type TABLET_DATA_TOMBSTONED (TS 1d34e18a1d394d3f91c9f50d1c7bea4e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:09.865518 8832 tablet_replica.cc:330] T 2aa1a5f266e840698f2127289628dcba P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:09.866605 8832 raft_consensus.cc:2229] T 2aa1a5f266e840698f2127289628dcba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:09.874528 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2aa1a5f266e840698f2127289628dcba with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:09.874146 8832 raft_consensus.cc:2258] T 2aa1a5f266e840698f2127289628dcba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:09.879573 8592 raft_consensus.cc:1050] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e: attempting to promote NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d to VOTER
I20240501 20:10:09.880627 8592 consensus_queue.cc:237] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:09.886437 8831 ts_tablet_manager.cc:902] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Initiating tablet copy from peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:09.891844 8829 tablet_replica.cc:330] T ca4f195a6dd14be981395538c2a33ad9 P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:09.897292 8829 raft_consensus.cc:2229] T ca4f195a6dd14be981395538c2a33ad9 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:09.897745 8829 raft_consensus.cc:2258] T ca4f195a6dd14be981395538c2a33ad9 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:09.898983 8829 ts_tablet_manager.cc:1875] T ca4f195a6dd14be981395538c2a33ad9 P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:09.912498 4091 catalog_manager.cc:5576] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817) does not have the latest schema for tablet 657466e8f25d4cc69ea61c80ffe9d210 (table rebalancer_dynamic_table [id=90b1b66130bb43bfa55a6b400ea31fd1]). Expected version 1 got 0
I20240501 20:10:09.917596 8829 ts_tablet_manager.cc:1888] T ca4f195a6dd14be981395538c2a33ad9 P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:09.917884 8829 log.cc:1196] T ca4f195a6dd14be981395538c2a33ad9 P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/ca4f195a6dd14be981395538c2a33ad9
I20240501 20:10:09.919924 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet ca4f195a6dd14be981395538c2a33ad9 (table rebalance_test_table_19 [id=8de1e355c457486da8bba4c628d52c17]) successfully deleted
I20240501 20:10:09.924722 8548 consensus_queue.cc:1035] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:09.925143 8838 tablet.cc:1722] T 657466e8f25d4cc69ea61c80ffe9d210 P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:09.924964 8832 ts_tablet_manager.cc:1875] T 2aa1a5f266e840698f2127289628dcba P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:09.926853 8838 tablet.cc:1620] T 657466e8f25d4cc69ea61c80ffe9d210 P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:09.927414 4083 catalog_manager.cc:5576] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237) does not have the latest schema for tablet 657466e8f25d4cc69ea61c80ffe9d210 (table rebalancer_dynamic_table [id=90b1b66130bb43bfa55a6b400ea31fd1]). Expected version 1 got 0
I20240501 20:10:09.930824 8839 tablet.cc:1722] T 657466e8f25d4cc69ea61c80ffe9d210 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:09.934722 7287 consensus_queue.cc:237] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:09.937897 8839 tablet.cc:1620] T 657466e8f25d4cc69ea61c80ffe9d210 P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:09.938758 8835 ts_tablet_manager.cc:902] T 6f2576acd7d34ec1ad27ab04a863df10 P 3e974c03d3c546f9ab6c91e762a5426b: Initiating tablet copy from peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:09.941551 8837 tablet.cc:1722] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:09.944437 8837 tablet.cc:1620] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:10:09.946625 7421 raft_consensus.cc:1261] T d1291a2b3dcc4c55a26df6a5cc390050 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:09.946650 7222 consensus_peers.cc:485] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1291a2b3dcc4c55a26df6a5cc390050. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:09.947495 4233 raft_consensus.cc:1261] T 1c6c89fa7e6f485e889a5197da4bd135 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:09.948240 8641 consensus_queue.cc:1035] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:09.948562 8784 consensus_queue.cc:1035] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:09.964129 4083 catalog_manager.cc:5576] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817) does not have the latest schema for tablet 657466e8f25d4cc69ea61c80ffe9d210 (table rebalancer_dynamic_table [id=90b1b66130bb43bfa55a6b400ea31fd1]). Expected version 1 got 0
I20240501 20:10:09.967533 7022 raft_consensus.cc:1261] T 1c6c89fa7e6f485e889a5197da4bd135 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:09.968536 8592 consensus_queue.cc:1035] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:09.970067 4087 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=90b1b66130bb43bfa55a6b400ea31fd1] alter complete (version 1)
I20240501 20:10:09.966938 4497 raft_consensus.cc:1261] T 1c6c89fa7e6f485e889a5197da4bd135 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:09.978600 8784 consensus_queue.cc:1035] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:09.983989 8532 consensus_queue.cc:1035] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:09.989873 8784 consensus_queue.cc:1035] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:09.998406 8641 raft_consensus.cc:2932] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:10.010794 7420 raft_consensus.cc:2932] T d1291a2b3dcc4c55a26df6a5cc390050 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:09.983628 7153 raft_consensus.cc:1261] T d1291a2b3dcc4c55a26df6a5cc390050 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:10.007846 8366 consensus_queue.cc:1035] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:10.034237 8842 consensus_queue.cc:1035] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:10.057775 4087 catalog_manager.cc:5484] T d1291a2b3dcc4c55a26df6a5cc390050 P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:10.063668 8139 consensus_queue.cc:1035] T 0e8f897c4d744c468b33b1326b0c366a P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:10.064791 8592 raft_consensus.cc:2932] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:10.072842 8832 ts_tablet_manager.cc:1888] T 2aa1a5f266e840698f2127289628dcba P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:10.073033 8832 log.cc:1196] T 2aa1a5f266e840698f2127289628dcba P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/2aa1a5f266e840698f2127289628dcba
I20240501 20:10:10.075513 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet 2aa1a5f266e840698f2127289628dcba (table rebalance_test_table_7 [id=59dd8401cbd048cea30daf86baac8cfa]) successfully deleted
I20240501 20:10:10.081490 8831 tablet_copy_client.cc:308] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.69:46237
I20240501 20:10:10.082968 7042 tablet_copy_service.cc:140] P ef0a4ada548446b3a531cb39a3c4e38d: Received BeginTabletCopySession request for tablet 3809cc8ffe6b4a9e99ebbb9d26a65279 from peer 1d34e18a1d394d3f91c9f50d1c7bea4e ({username='slave'} at 127.29.219.68:40675)
I20240501 20:10:10.083149 7042 tablet_copy_service.cc:161] P ef0a4ada548446b3a531cb39a3c4e38d: Beginning new tablet copy session on tablet 3809cc8ffe6b4a9e99ebbb9d26a65279 from peer 1d34e18a1d394d3f91c9f50d1c7bea4e at {username='slave'} at 127.29.219.68:40675: session id = 1d34e18a1d394d3f91c9f50d1c7bea4e-3809cc8ffe6b4a9e99ebbb9d26a65279
I20240501 20:10:10.083164 8835 tablet_copy_client.cc:308] T 6f2576acd7d34ec1ad27ab04a863df10 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.68:45205
I20240501 20:10:10.084025 4651 tablet_copy_service.cc:140] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Received BeginTabletCopySession request for tablet 6f2576acd7d34ec1ad27ab04a863df10 from peer 3e974c03d3c546f9ab6c91e762a5426b ({username='slave'} at 127.29.219.67:55693)
I20240501 20:10:10.084175 4651 tablet_copy_service.cc:161] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Beginning new tablet copy session on tablet 6f2576acd7d34ec1ad27ab04a863df10 from peer 3e974c03d3c546f9ab6c91e762a5426b at {username='slave'} at 127.29.219.67:55693: session id = 3e974c03d3c546f9ab6c91e762a5426b-6f2576acd7d34ec1ad27ab04a863df10
I20240501 20:10:10.085884 7042 tablet_copy_source_session.cc:205] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:10.086239 4651 tablet_copy_source_session.cc:205] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:10.087654 8314 consensus_queue.cc:1035] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:10.087946 8835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f2576acd7d34ec1ad27ab04a863df10. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:10.087857 4233 raft_consensus.cc:2932] T 1c6c89fa7e6f485e889a5197da4bd135 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:10.090088 4083 catalog_manager.cc:5484] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:10.075592 7153 raft_consensus.cc:2932] T d1291a2b3dcc4c55a26df6a5cc390050 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:10.096487 4631 consensus_queue.cc:237] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:10.099123 8831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3809cc8ffe6b4a9e99ebbb9d26a65279. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:10.101598 4497 raft_consensus.cc:1261] T 1c6c89fa7e6f485e889a5197da4bd135 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:10.103919 7022 raft_consensus.cc:2932] T 1c6c89fa7e6f485e889a5197da4bd135 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:10.105530 8314 consensus_queue.cc:1035] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:10.106595 4083 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:10.106474 4498 raft_consensus.cc:2932] T 1c6c89fa7e6f485e889a5197da4bd135 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:10.109548 8831 tablet_copy_client.cc:783] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:10.109804 8831 tablet_copy_client.cc:647] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:10.113276 8831 tablet_copy_client.cc:523] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:10.118968 8831 tablet_bootstrap.cc:492] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:10.119256 8839 tablet.cc:1722] T 657466e8f25d4cc69ea61c80ffe9d210 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:10.119483 8838 tablet.cc:1722] T 657466e8f25d4cc69ea61c80ffe9d210 P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:10.119781 8839 tablet.cc:1620] T 657466e8f25d4cc69ea61c80ffe9d210 P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:10.119956 8838 tablet.cc:1620] T 657466e8f25d4cc69ea61c80ffe9d210 P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:10.128023 8314 raft_consensus.cc:2932] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:10.128425 7020 raft_consensus.cc:1261] T 1c6c89fa7e6f485e889a5197da4bd135 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:10.131533 8374 consensus_queue.cc:1035] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:10.132162 8837 tablet.cc:1722] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:10.132790 8837 tablet.cc:1620] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:10:10.132542 4497 raft_consensus.cc:2932] T 1c6c89fa7e6f485e889a5197da4bd135 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:10.136162 7020 raft_consensus.cc:2932] T 1c6c89fa7e6f485e889a5197da4bd135 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:10.144436 4090 catalog_manager.cc:5484] T 1c6c89fa7e6f485e889a5197da4bd135 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 5 to 6, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:10.144927 4079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1c6c89fa7e6f485e889a5197da4bd135 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:10.161967 8835 tablet_copy_client.cc:783] T 6f2576acd7d34ec1ad27ab04a863df10 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:10.162323 8835 tablet_copy_client.cc:647] T 6f2576acd7d34ec1ad27ab04a863df10 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:10.163874 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c6c89fa7e6f485e889a5197da4bd135 with delete_type TABLET_DATA_TOMBSTONED (TS dca57823003e4729b769ca12367d7530 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:10.165601 8871 tablet_replica.cc:330] T 1c6c89fa7e6f485e889a5197da4bd135 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:10.166106 8871 raft_consensus.cc:2229] T 1c6c89fa7e6f485e889a5197da4bd135 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:10.166159 8835 tablet_copy_client.cc:523] T 6f2576acd7d34ec1ad27ab04a863df10 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:10.166471 8871 raft_consensus.cc:2258] T 1c6c89fa7e6f485e889a5197da4bd135 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:10.167939 8871 ts_tablet_manager.cc:1875] T 1c6c89fa7e6f485e889a5197da4bd135 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:10.170817 8835 tablet_bootstrap.cc:492] T 6f2576acd7d34ec1ad27ab04a863df10 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:10.171754 4087 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=90b1b66130bb43bfa55a6b400ea31fd1] alter complete (version 2)
I20240501 20:10:10.175608 8871 ts_tablet_manager.cc:1888] T 1c6c89fa7e6f485e889a5197da4bd135 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:10.175817 8871 log.cc:1196] T 1c6c89fa7e6f485e889a5197da4bd135 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/1c6c89fa7e6f485e889a5197da4bd135
I20240501 20:10:10.176724 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet 1c6c89fa7e6f485e889a5197da4bd135 (table rebalance_test_table_11 [id=3b02a4c6c0f54681a87f5ca2381de01d]) successfully deleted
I20240501 20:10:10.179651 8835 tablet_bootstrap.cc:492] T 6f2576acd7d34ec1ad27ab04a863df10 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:10.180120 8835 tablet_bootstrap.cc:492] T 6f2576acd7d34ec1ad27ab04a863df10 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap complete.
I20240501 20:10:10.180354 8835 ts_tablet_manager.cc:1367] T 6f2576acd7d34ec1ad27ab04a863df10 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240501 20:10:10.181361 8835 raft_consensus.cc:348] T 6f2576acd7d34ec1ad27ab04a863df10 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:10.181604 8835 raft_consensus.cc:729] T 6f2576acd7d34ec1ad27ab04a863df10 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: LEARNER
I20240501 20:10:10.181860 8835 consensus_queue.cc:260] T 6f2576acd7d34ec1ad27ab04a863df10 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:10.183504 8835 ts_tablet_manager.cc:1398] T 6f2576acd7d34ec1ad27ab04a863df10 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:10:10.184147 4651 tablet_copy_service.cc:342] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Request end of tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-6f2576acd7d34ec1ad27ab04a863df10 received from {username='slave'} at 127.29.219.67:55693
I20240501 20:10:10.184260 4651 tablet_copy_service.cc:433] P 1d34e18a1d394d3f91c9f50d1c7bea4e: ending tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-6f2576acd7d34ec1ad27ab04a863df10 on tablet 6f2576acd7d34ec1ad27ab04a863df10 with peer 3e974c03d3c546f9ab6c91e762a5426b
I20240501 20:10:10.190121 4233 consensus_queue.cc:237] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:10.203742 8831 tablet_bootstrap.cc:492] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:10.204321 8831 tablet_bootstrap.cc:492] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap complete.
I20240501 20:10:10.204578 8831 ts_tablet_manager.cc:1367] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.086s user 0.004s sys 0.003s
I20240501 20:10:10.206298 8831 raft_consensus.cc:348] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:10.206552 8831 raft_consensus.cc:729] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: LEARNER
I20240501 20:10:10.206797 8831 consensus_queue.cc:260] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:10.210907 8831 ts_tablet_manager.cc:1398] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
W20240501 20:10:10.210937 4169 consensus_peers.cc:485] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f18af954cefa48d4882313fe18abbd27. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:10.211817 4497 raft_consensus.cc:1261] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:10.211858 4091 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:10.212476 4631 raft_consensus.cc:1261] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:10.212524 8532 consensus_queue.cc:1035] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:10.216387 8532 consensus_queue.cc:1035] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:10.222255 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 9e31f8d1809244efae05ab9254c163ce (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=880fdf37926743948f9ffe03bdaeeaa5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:10.222795 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e31f8d1809244efae05ab9254c163ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:10.223587 7266 tablet_service.cc:1467] Processing CreateTablet for tablet 9e31f8d1809244efae05ab9254c163ce (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=880fdf37926743948f9ffe03bdaeeaa5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:10.224012 7266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e31f8d1809244efae05ab9254c163ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:10.226290 7135 tablet_service.cc:1467] Processing CreateTablet for tablet 48528ec242974de4ac20eb0161f0cb05 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=880fdf37926743948f9ffe03bdaeeaa5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:10.227308 8532 raft_consensus.cc:2932] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:10.227645 7267 tablet_service.cc:1467] Processing CreateTablet for tablet ce89f9b962264db7a91685556f1354a8 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=880fdf37926743948f9ffe03bdaeeaa5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:10.228087 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce89f9b962264db7a91685556f1354a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:10.230697 4498 raft_consensus.cc:2932] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:10.232662 7000 tablet_service.cc:1467] Processing CreateTablet for tablet ce89f9b962264db7a91685556f1354a8 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=880fdf37926743948f9ffe03bdaeeaa5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:10.233119 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce89f9b962264db7a91685556f1354a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:10.233724 4347 tablet_service.cc:1467] Processing CreateTablet for tablet ce89f9b962264db7a91685556f1354a8 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=880fdf37926743948f9ffe03bdaeeaa5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:10.234179 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce89f9b962264db7a91685556f1354a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:10.235021 7001 tablet_service.cc:1467] Processing CreateTablet for tablet 48528ec242974de4ac20eb0161f0cb05 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=880fdf37926743948f9ffe03bdaeeaa5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:10.235299 7135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48528ec242974de4ac20eb0161f0cb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:10.235390 7001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48528ec242974de4ac20eb0161f0cb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:10.239006 7400 tablet_service.cc:1467] Processing CreateTablet for tablet 48528ec242974de4ac20eb0161f0cb05 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=880fdf37926743948f9ffe03bdaeeaa5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:10.239477 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48528ec242974de4ac20eb0161f0cb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:10.241781 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 9e31f8d1809244efae05ab9254c163ce (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=880fdf37926743948f9ffe03bdaeeaa5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:10.242230 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e31f8d1809244efae05ab9254c163ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:10.248029 4233 consensus_queue.cc:237] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:10.251017 7022 raft_consensus.cc:1261] T 644d4d293e8c462297c6419965bd2c89 P ef0a4ada548446b3a531cb39a3c4e38d [term 2 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:10:10.251175 4365 raft_consensus.cc:1261] T 644d4d293e8c462297c6419965bd2c89 P 10b47782558c41ee91f5e164a372049a [term 2 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:10:10.253156 8785 consensus_queue.cc:1035] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:10.253837 8790 consensus_queue.cc:1035] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:10.258491 7042 tablet_copy_service.cc:342] P ef0a4ada548446b3a531cb39a3c4e38d: Request end of tablet copy session 1d34e18a1d394d3f91c9f50d1c7bea4e-3809cc8ffe6b4a9e99ebbb9d26a65279 received from {username='slave'} at 127.29.219.68:40675
I20240501 20:10:10.258751 7042 tablet_copy_service.cc:433] P ef0a4ada548446b3a531cb39a3c4e38d: ending tablet copy session 1d34e18a1d394d3f91c9f50d1c7bea4e-3809cc8ffe6b4a9e99ebbb9d26a65279 on tablet 3809cc8ffe6b4a9e99ebbb9d26a65279 with peer 1d34e18a1d394d3f91c9f50d1c7bea4e
I20240501 20:10:10.261126 4084 catalog_manager.cc:5484] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:10.262599 8790 raft_consensus.cc:2932] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:10.268182 7022 raft_consensus.cc:2932] T 644d4d293e8c462297c6419965bd2c89 P ef0a4ada548446b3a531cb39a3c4e38d [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:10.269315 8879 tablet_bootstrap.cc:492] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:10.271353 8879 tablet_bootstrap.cc:654] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:10.271246 4365 raft_consensus.cc:2932] T 644d4d293e8c462297c6419965bd2c89 P 10b47782558c41ee91f5e164a372049a [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:10.274571 4091 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:10.278491 4087 catalog_manager.cc:5484] T 644d4d293e8c462297c6419965bd2c89 P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index 5 to 7, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New cstate: current_term: 2 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:10.284507 4498 raft_consensus.cc:1203] T 6f2576acd7d34ec1ad27ab04a863df10 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:10.285264 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 644d4d293e8c462297c6419965bd2c89 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:10.289480 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet 644d4d293e8c462297c6419965bd2c89 with delete_type TABLET_DATA_TOMBSTONED (TS 3e974c03d3c546f9ab6c91e762a5426b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:10.289916 8839 tablet.cc:1722] T 657466e8f25d4cc69ea61c80ffe9d210 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:10.288949 8837 tablet.cc:1722] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:10.301527 8838 tablet.cc:1722] T 657466e8f25d4cc69ea61c80ffe9d210 P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:10.303385 4087 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table_new_name [id=90b1b66130bb43bfa55a6b400ea31fd1] alter complete (version 3)
I20240501 20:10:10.305678 8880 tablet_bootstrap.cc:492] T 9e31f8d1809244efae05ab9254c163ce P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:10.307638 8880 tablet_bootstrap.cc:654] T 9e31f8d1809244efae05ab9254c163ce P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:10.332203 8878 ts_tablet_manager.cc:902] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5: Initiating tablet copy from peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:10.333247 8883 tablet_bootstrap.cc:492] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:10.335477 8883 tablet_bootstrap.cc:654] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:10.339375 4627 raft_consensus.cc:2932] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:10.344262 8891 tablet_bootstrap.cc:492] T ce89f9b962264db7a91685556f1354a8 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:10.345896 8885 tablet_replica.cc:330] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:10.346539 8891 tablet_bootstrap.cc:654] T ce89f9b962264db7a91685556f1354a8 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:10.348719 8882 tablet_bootstrap.cc:492] T 9e31f8d1809244efae05ab9254c163ce P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:10.342311 8879 tablet_bootstrap.cc:492] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:10.350648 8885 raft_consensus.cc:2229] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:10.350843 8882 tablet_bootstrap.cc:654] T 9e31f8d1809244efae05ab9254c163ce P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:10.350929 8885 raft_consensus.cc:2258] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:10.351457 8879 ts_tablet_manager.cc:1367] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.082s user 0.006s sys 0.000s
I20240501 20:10:10.351989 8885 ts_tablet_manager.cc:1875] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:10.352921 8880 tablet_bootstrap.cc:492] T 9e31f8d1809244efae05ab9254c163ce P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:10.354082 8881 tablet_bootstrap.cc:492] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:10.354656 8889 ts_tablet_manager.cc:902] T d1291a2b3dcc4c55a26df6a5cc390050 P 10b47782558c41ee91f5e164a372049a: Initiating tablet copy from peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:10.356313 8881 tablet_bootstrap.cc:654] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:10.361778 8885 ts_tablet_manager.cc:1888] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:10:10.361994 8885 log.cc:1196] T 644d4d293e8c462297c6419965bd2c89 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/644d4d293e8c462297c6419965bd2c89
I20240501 20:10:10.363070 8880 ts_tablet_manager.cc:1367] T 9e31f8d1809244efae05ab9254c163ce P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.058s user 0.000s sys 0.004s
I20240501 20:10:10.363446 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet 644d4d293e8c462297c6419965bd2c89 (table rebalance_test_table_9 [id=eef6790ef9874038b63d81f758f188d7]) successfully deleted
I20240501 20:10:10.363672 8879 raft_consensus.cc:348] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:10.363997 8879 raft_consensus.cc:374] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:10.365208 8880 raft_consensus.cc:348] T 9e31f8d1809244efae05ab9254c163ce P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:10.365516 8880 raft_consensus.cc:374] T 9e31f8d1809244efae05ab9254c163ce P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:10.365789 8883 tablet_bootstrap.cc:492] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:10.366137 8883 ts_tablet_manager.cc:1367] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.004s
I20240501 20:10:10.366645 8879 raft_consensus.cc:729] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:10.367019 8883 raft_consensus.cc:348] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:10.367594 8883 raft_consensus.cc:374] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:10.367786 8883 raft_consensus.cc:729] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:10.367575 8879 consensus_queue.cc:260] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:10.368157 8880 raft_consensus.cc:729] T 9e31f8d1809244efae05ab9254c163ce P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:10.368230 8883 consensus_queue.cc:260] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:10.371758 8883 ts_tablet_manager.cc:1398] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:10:10.378530 8879 ts_tablet_manager.cc:1398] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20240501 20:10:10.379027 8880 consensus_queue.cc:260] T 9e31f8d1809244efae05ab9254c163ce P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:10.383543 8882 tablet_bootstrap.cc:492] T 9e31f8d1809244efae05ab9254c163ce P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:10.383845 8882 ts_tablet_manager.cc:1367] T 9e31f8d1809244efae05ab9254c163ce P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.003s
I20240501 20:10:10.385136 8882 raft_consensus.cc:348] T 9e31f8d1809244efae05ab9254c163ce P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:10.385408 8880 ts_tablet_manager.cc:1398] T 9e31f8d1809244efae05ab9254c163ce P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.021s user 0.000s sys 0.002s
I20240501 20:10:10.385594 8882 raft_consensus.cc:374] T 9e31f8d1809244efae05ab9254c163ce P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:10.385798 8882 raft_consensus.cc:729] T 9e31f8d1809244efae05ab9254c163ce P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:10.392244 4631 raft_consensus.cc:1203] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:10.400490 8881 tablet_bootstrap.cc:492] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:10:10.400813 8881 ts_tablet_manager.cc:1367] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.000s
I20240501 20:10:10.404806 8891 tablet_bootstrap.cc:492] T ce89f9b962264db7a91685556f1354a8 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:10.405079 8891 ts_tablet_manager.cc:1367] T ce89f9b962264db7a91685556f1354a8 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.061s user 0.006s sys 0.000s
I20240501 20:10:10.405982 8891 raft_consensus.cc:348] T ce89f9b962264db7a91685556f1354a8 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:10.406352 8891 raft_consensus.cc:374] T ce89f9b962264db7a91685556f1354a8 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:10.406560 8891 raft_consensus.cc:729] T ce89f9b962264db7a91685556f1354a8 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:10.407030 8891 consensus_queue.cc:260] T ce89f9b962264db7a91685556f1354a8 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:10.413798 8883 tablet_bootstrap.cc:492] T 48528ec242974de4ac20eb0161f0cb05 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:10.416636 8882 consensus_queue.cc:260] T 9e31f8d1809244efae05ab9254c163ce P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:10.419437 8882 ts_tablet_manager.cc:1398] T 9e31f8d1809244efae05ab9254c163ce P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.035s user 0.002s sys 0.001s
I20240501 20:10:10.402491 8881 raft_consensus.cc:348] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:10.420433 8881 raft_consensus.cc:374] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:10.420717 8881 raft_consensus.cc:729] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:10:10.421183 8881 consensus_queue.cc:260] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:10.429028 8891 ts_tablet_manager.cc:1398] T ce89f9b962264db7a91685556f1354a8 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240501 20:10:10.430069 8795 raft_consensus.cc:482] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:10.430236 8795 raft_consensus.cc:504] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:10.430939 8881 ts_tablet_manager.cc:1398] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.030s user 0.003s sys 0.000s
I20240501 20:10:10.431324 8878 tablet_copy_client.cc:308] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.67:34595
I20240501 20:10:10.431417 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce89f9b962264db7a91685556f1354a8" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:10.431824 8883 tablet_bootstrap.cc:654] T 48528ec242974de4ac20eb0161f0cb05 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:10.432283 4519 tablet_copy_service.cc:140] P 3e974c03d3c546f9ab6c91e762a5426b: Received BeginTabletCopySession request for tablet ffaf0a0105ea4e1f9ed9d90c221ebc8a from peer b9957d4086bc4a76b18ed9265a5739d5 ({username='slave'} at 127.29.219.72:58885)
I20240501 20:10:10.432467 4519 tablet_copy_service.cc:161] P 3e974c03d3c546f9ab6c91e762a5426b: Beginning new tablet copy session on tablet ffaf0a0105ea4e1f9ed9d90c221ebc8a from peer b9957d4086bc4a76b18ed9265a5739d5 at {username='slave'} at 127.29.219.72:58885: session id = b9957d4086bc4a76b18ed9265a5739d5-ffaf0a0105ea4e1f9ed9d90c221ebc8a
I20240501 20:10:10.434370 8882 tablet_bootstrap.cc:492] T ce89f9b962264db7a91685556f1354a8 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:10.434947 4087 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:10.435173 4087 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:10.436379 8882 tablet_bootstrap.cc:654] T ce89f9b962264db7a91685556f1354a8 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:10.431733 8795 leader_election.cc:288] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
W20240501 20:10:10.432457 4300 leader_election.cc:341] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:10.441380 4519 tablet_copy_source_session.cc:205] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:10.442270 4087 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 657466e8f25d4cc69ea61c80ffe9d210
I20240501 20:10:10.442785 8889 tablet_copy_client.cc:308] T d1291a2b3dcc4c55a26df6a5cc390050 P 10b47782558c41ee91f5e164a372049a: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.71:39447
I20240501 20:10:10.443791 7308 tablet_copy_service.cc:140] P bceeae1c79964acabdae8ff771bf0c97: Received BeginTabletCopySession request for tablet d1291a2b3dcc4c55a26df6a5cc390050 from peer 10b47782558c41ee91f5e164a372049a ({username='slave'} at 127.29.219.66:32801)
I20240501 20:10:10.443933 7308 tablet_copy_service.cc:161] P bceeae1c79964acabdae8ff771bf0c97: Beginning new tablet copy session on tablet d1291a2b3dcc4c55a26df6a5cc390050 from peer 10b47782558c41ee91f5e164a372049a at {username='slave'} at 127.29.219.66:32801: session id = 10b47782558c41ee91f5e164a372049a-d1291a2b3dcc4c55a26df6a5cc390050
I20240501 20:10:10.444862 7135 tablet_service.cc:1514] Processing DeleteTablet for tablet 657466e8f25d4cc69ea61c80ffe9d210 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:10 UTC) from {username='slave'} at 127.0.0.1:52036
I20240501 20:10:10.446010 8878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffaf0a0105ea4e1f9ed9d90c221ebc8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:10.448606 7308 tablet_copy_source_session.cc:205] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:10.450758 7401 tablet_service.cc:1514] Processing DeleteTablet for tablet 657466e8f25d4cc69ea61c80ffe9d210 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:10 UTC) from {username='slave'} at 127.0.0.1:40182
I20240501 20:10:10.434846 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce89f9b962264db7a91685556f1354a8" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:10.454067 7022 raft_consensus.cc:2454] T ce89f9b962264db7a91685556f1354a8 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:10:10.443393 7000 tablet_service.cc:1514] Processing DeleteTablet for tablet 657466e8f25d4cc69ea61c80ffe9d210 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:10 UTC) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:10.455307 4301 leader_election.cc:302] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, ef0a4ada548446b3a531cb39a3c4e38d; no voters: bceeae1c79964acabdae8ff771bf0c97
I20240501 20:10:10.456084 8795 raft_consensus.cc:2781] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:10.456240 8795 raft_consensus.cc:482] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:10.456364 8795 raft_consensus.cc:3037] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:10.457621 8851 raft_consensus.cc:482] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:10.457774 8851 raft_consensus.cc:504] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:10.458092 8889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1291a2b3dcc4c55a26df6a5cc390050. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:10.459525 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e31f8d1809244efae05ab9254c163ce" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:10.459822 4233 raft_consensus.cc:2454] T 9e31f8d1809244efae05ab9254c163ce P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:10:10.459815 8795 raft_consensus.cc:504] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:10.460263 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e31f8d1809244efae05ab9254c163ce" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:10.460520 7287 raft_consensus.cc:2454] T 9e31f8d1809244efae05ab9254c163ce P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:10:10.460884 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce89f9b962264db7a91685556f1354a8" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:10.461077 7022 raft_consensus.cc:3037] T ce89f9b962264db7a91685556f1354a8 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:10.461004 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce89f9b962264db7a91685556f1354a8" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
W20240501 20:10:10.461658 4300 leader_election.cc:341] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:10.462255 7354 leader_election.cc:302] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:10.462569 8430 raft_consensus.cc:2781] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:10.462683 8430 raft_consensus.cc:482] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:10.462790 8430 raft_consensus.cc:3037] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:10.464062 7022 raft_consensus.cc:2454] T ce89f9b962264db7a91685556f1354a8 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:10:10.464479 4301 leader_election.cc:302] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, ef0a4ada548446b3a531cb39a3c4e38d; no voters: bceeae1c79964acabdae8ff771bf0c97
I20240501 20:10:10.466220 8795 leader_election.cc:288] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Requested vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:10.466588 8795 raft_consensus.cc:2781] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:10.466751 8795 raft_consensus.cc:686] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Becoming Leader. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Running, Role: LEADER
I20240501 20:10:10.467056 8795 consensus_queue.cc:237] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:10.465927 8430 raft_consensus.cc:504] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:10.472872 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e31f8d1809244efae05ab9254c163ce" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:10:10.473194 4233 raft_consensus.cc:3037] T 9e31f8d1809244efae05ab9254c163ce P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:10.468513 8911 tablet_replica.cc:330] T 657466e8f25d4cc69ea61c80ffe9d210 P b9957d4086bc4a76b18ed9265a5739d5: stopping tablet replica
I20240501 20:10:10.473448 8911 raft_consensus.cc:2229] T 657466e8f25d4cc69ea61c80ffe9d210 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:10.473723 8911 raft_consensus.cc:2258] T 657466e8f25d4cc69ea61c80ffe9d210 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:10.468777 8851 leader_election.cc:288] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:10.476399 8911 ts_tablet_manager.cc:1875] T 657466e8f25d4cc69ea61c80ffe9d210 P b9957d4086bc4a76b18ed9265a5739d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:10.477080 4233 raft_consensus.cc:2454] T 9e31f8d1809244efae05ab9254c163ce P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:10.478052 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e31f8d1809244efae05ab9254c163ce" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:10.478338 7287 raft_consensus.cc:3037] T 9e31f8d1809244efae05ab9254c163ce P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:10.478699 8891 tablet_bootstrap.cc:492] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:10.480563 8891 tablet_bootstrap.cc:654] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:10.481220 7287 raft_consensus.cc:2454] T 9e31f8d1809244efae05ab9254c163ce P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:10.481784 7354 leader_election.cc:302] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:10.482239 8851 raft_consensus.cc:2781] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:10.482700 8430 leader_election.cc:288] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Requested vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:10.484416 8911 ts_tablet_manager.cc:1888] T 657466e8f25d4cc69ea61c80ffe9d210 P b9957d4086bc4a76b18ed9265a5739d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:10.487287 8889 tablet_copy_client.cc:783] T d1291a2b3dcc4c55a26df6a5cc390050 P 10b47782558c41ee91f5e164a372049a: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:10.487844 8889 tablet_copy_client.cc:647] T d1291a2b3dcc4c55a26df6a5cc390050 P 10b47782558c41ee91f5e164a372049a: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:10.490345 8889 tablet_copy_client.cc:523] T d1291a2b3dcc4c55a26df6a5cc390050 P 10b47782558c41ee91f5e164a372049a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:10.494524 8912 tablet_replica.cc:330] T 657466e8f25d4cc69ea61c80ffe9d210 P ef0a4ada548446b3a531cb39a3c4e38d: stopping tablet replica
I20240501 20:10:10.494792 8912 raft_consensus.cc:2229] T 657466e8f25d4cc69ea61c80ffe9d210 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:10.495121 8912 raft_consensus.cc:2258] T 657466e8f25d4cc69ea61c80ffe9d210 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:10.496277 8912 ts_tablet_manager.cc:1875] T 657466e8f25d4cc69ea61c80ffe9d210 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:10.495337 8851 raft_consensus.cc:686] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Becoming Leader. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Running, Role: LEADER
I20240501 20:10:10.500253 8911 log.cc:1196] T 657466e8f25d4cc69ea61c80ffe9d210 P b9957d4086bc4a76b18ed9265a5739d5: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/wals/657466e8f25d4cc69ea61c80ffe9d210
I20240501 20:10:10.500353 8851 consensus_queue.cc:237] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:10.501133 8911 ts_tablet_manager.cc:1909] T 657466e8f25d4cc69ea61c80ffe9d210 P b9957d4086bc4a76b18ed9265a5739d5: Deleting consensus metadata
I20240501 20:10:10.502971 8889 tablet_bootstrap.cc:492] T d1291a2b3dcc4c55a26df6a5cc390050 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:10.503367 8883 tablet_bootstrap.cc:492] T 48528ec242974de4ac20eb0161f0cb05 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:10.503631 8883 ts_tablet_manager.cc:1367] T 48528ec242974de4ac20eb0161f0cb05 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.090s user 0.006s sys 0.000s
I20240501 20:10:10.503875 4087 catalog_manager.cc:5484] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a reported cstate change: term changed from 0 to 1, leader changed from <none> to 10b47782558c41ee91f5e164a372049a (127.29.219.66). New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:10.505478 8882 tablet_bootstrap.cc:492] T ce89f9b962264db7a91685556f1354a8 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:10.505712 8882 ts_tablet_manager.cc:1367] T ce89f9b962264db7a91685556f1354a8 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.086s user 0.004s sys 0.000s
I20240501 20:10:10.506603 4079 catalog_manager.cc:4830] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): tablet 657466e8f25d4cc69ea61c80ffe9d210 (table rebalancer_dynamic_table_new_name [id=90b1b66130bb43bfa55a6b400ea31fd1]) successfully deleted
I20240501 20:10:10.506809 8882 raft_consensus.cc:348] T ce89f9b962264db7a91685556f1354a8 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:10.507030 8882 raft_consensus.cc:374] T ce89f9b962264db7a91685556f1354a8 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:10.507091 8882 raft_consensus.cc:729] T ce89f9b962264db7a91685556f1354a8 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:10.507309 8882 consensus_queue.cc:260] T ce89f9b962264db7a91685556f1354a8 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:10.509039 8882 ts_tablet_manager.cc:1398] T ce89f9b962264db7a91685556f1354a8 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:10.516188 8891 tablet_bootstrap.cc:492] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:10.516520 8891 ts_tablet_manager.cc:1367] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20240501 20:10:10.517375 8891 raft_consensus.cc:348] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:10.519937 8883 raft_consensus.cc:348] T 48528ec242974de4ac20eb0161f0cb05 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:10.520331 8883 raft_consensus.cc:374] T 48528ec242974de4ac20eb0161f0cb05 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:10.520449 8883 raft_consensus.cc:729] T 48528ec242974de4ac20eb0161f0cb05 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:10.520982 8883 consensus_queue.cc:260] T 48528ec242974de4ac20eb0161f0cb05 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:10.521800 8883 ts_tablet_manager.cc:1398] T 48528ec242974de4ac20eb0161f0cb05 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240501 20:10:10.523517 8891 raft_consensus.cc:374] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:10.523649 8891 raft_consensus.cc:729] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:10.523972 8891 consensus_queue.cc:260] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:10.526513 8878 tablet_copy_client.cc:783] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:10.526798 8878 tablet_copy_client.cc:647] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:10.527550 8891 ts_tablet_manager.cc:1398] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 20:10:10.528378 8912 ts_tablet_manager.cc:1888] T 657466e8f25d4cc69ea61c80ffe9d210 P ef0a4ada548446b3a531cb39a3c4e38d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:10.530472 8878 tablet_copy_client.cc:523] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:10.532718 8889 tablet_bootstrap.cc:492] T d1291a2b3dcc4c55a26df6a5cc390050 P 10b47782558c41ee91f5e164a372049a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:10.532968 8912 log.cc:1196] T 657466e8f25d4cc69ea61c80ffe9d210 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/wals/657466e8f25d4cc69ea61c80ffe9d210
I20240501 20:10:10.533334 8889 tablet_bootstrap.cc:492] T d1291a2b3dcc4c55a26df6a5cc390050 P 10b47782558c41ee91f5e164a372049a: Bootstrap complete.
I20240501 20:10:10.533612 8889 ts_tablet_manager.cc:1367] T d1291a2b3dcc4c55a26df6a5cc390050 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.000s
I20240501 20:10:10.533668 8912 ts_tablet_manager.cc:1909] T 657466e8f25d4cc69ea61c80ffe9d210 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting consensus metadata
I20240501 20:10:10.534574 8889 raft_consensus.cc:348] T d1291a2b3dcc4c55a26df6a5cc390050 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:10.534811 8910 tablet_replica.cc:330] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd: stopping tablet replica
I20240501 20:10:10.534888 8889 raft_consensus.cc:729] T d1291a2b3dcc4c55a26df6a5cc390050 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: LEARNER
I20240501 20:10:10.535072 8910 raft_consensus.cc:2229] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:10.535120 8889 consensus_queue.cc:260] T d1291a2b3dcc4c55a26df6a5cc390050 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:10.535483 8910 raft_consensus.cc:2258] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:10.537892 8889 ts_tablet_manager.cc:1398] T d1291a2b3dcc4c55a26df6a5cc390050 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:10:10.537906 8910 ts_tablet_manager.cc:1875] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:10.537886 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet 657466e8f25d4cc69ea61c80ffe9d210 (table rebalancer_dynamic_table_new_name [id=90b1b66130bb43bfa55a6b400ea31fd1]) successfully deleted
I20240501 20:10:10.538884 7308 tablet_copy_service.cc:342] P bceeae1c79964acabdae8ff771bf0c97: Request end of tablet copy session 10b47782558c41ee91f5e164a372049a-d1291a2b3dcc4c55a26df6a5cc390050 received from {username='slave'} at 127.29.219.66:32801
I20240501 20:10:10.539057 7308 tablet_copy_service.cc:433] P bceeae1c79964acabdae8ff771bf0c97: ending tablet copy session 10b47782558c41ee91f5e164a372049a-d1291a2b3dcc4c55a26df6a5cc390050 on tablet d1291a2b3dcc4c55a26df6a5cc390050 with peer 10b47782558c41ee91f5e164a372049a
I20240501 20:10:10.537134 4087 catalog_manager.cc:5484] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72). New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:10.541810 4233 consensus_queue.cc:237] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:10.551134 8910 ts_tablet_manager.cc:1888] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:10.551376 8910 log.cc:1196] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/wals/657466e8f25d4cc69ea61c80ffe9d210
I20240501 20:10:10.551873 8910 ts_tablet_manager.cc:1909] T 657466e8f25d4cc69ea61c80ffe9d210 P db0742ee3860468590eefef6ae6074fd: Deleting consensus metadata
I20240501 20:10:10.552273 8878 tablet_bootstrap.cc:492] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:10.553074 4076 catalog_manager.cc:4830] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): tablet 657466e8f25d4cc69ea61c80ffe9d210 (table rebalancer_dynamic_table_new_name [id=90b1b66130bb43bfa55a6b400ea31fd1]) successfully deleted
W20240501 20:10:10.554234 4169 consensus_peers.cc:485] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f44675b6874a423398de361c7ccfd583. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:10.555341 7287 raft_consensus.cc:1261] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:10.555220 4087 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:10.555882 4364 raft_consensus.cc:1261] T f44675b6874a423398de361c7ccfd583 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:10.556090 8532 consensus_queue.cc:1035] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:10.557112 8532 consensus_queue.cc:1035] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:10.560382 8532 raft_consensus.cc:2932] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:10.563596 7287 raft_consensus.cc:2932] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:10.567229 7000 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd491a4c3dd42f7910ae61bf68537c3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7274550d79fa46a59cd847795c41e629]), partition=
I20240501 20:10:10.567328 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd491a4c3dd42f7910ae61bf68537c3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7274550d79fa46a59cd847795c41e629]), partition=
I20240501 20:10:10.567649 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd491a4c3dd42f7910ae61bf68537c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:10.567728 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd491a4c3dd42f7910ae61bf68537c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:10.571838 7266 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd491a4c3dd42f7910ae61bf68537c3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7274550d79fa46a59cd847795c41e629]), partition=
I20240501 20:10:10.572237 7266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd491a4c3dd42f7910ae61bf68537c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:10.573786 8883 tablet_bootstrap.cc:492] T 0dd491a4c3dd42f7910ae61bf68537c3 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:10.575490 8883 tablet_bootstrap.cc:654] T 0dd491a4c3dd42f7910ae61bf68537c3 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:10.576373 8891 tablet_bootstrap.cc:492] T 0dd491a4c3dd42f7910ae61bf68537c3 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:10.577912 8891 tablet_bootstrap.cc:654] T 0dd491a4c3dd42f7910ae61bf68537c3 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:10.587424 8882 tablet_bootstrap.cc:492] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:10.589087 8882 tablet_bootstrap.cc:654] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:10.589315 4087 catalog_manager.cc:5484] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:10.599831 8495 raft_consensus.cc:482] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:10.600044 8495 raft_consensus.cc:504] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:10.600744 8495 leader_election.cc:288] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:10.601512 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48528ec242974de4ac20eb0161f0cb05" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:10.601783 7155 raft_consensus.cc:2454] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:10.602300 6955 leader_election.cc:302] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db0742ee3860468590eefef6ae6074fd, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:10.602581 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48528ec242974de4ac20eb0161f0cb05" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:10.602952 8877 raft_consensus.cc:2781] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:10.603013 7420 raft_consensus.cc:2454] T 48528ec242974de4ac20eb0161f0cb05 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:10.603108 8877 raft_consensus.cc:482] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:10.603212 8877 raft_consensus.cc:3037] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:10.606117 8877 raft_consensus.cc:504] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:10.607836 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48528ec242974de4ac20eb0161f0cb05" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:10.608080 7420 raft_consensus.cc:3037] T 48528ec242974de4ac20eb0161f0cb05 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:10.608924 4364 raft_consensus.cc:2932] T f44675b6874a423398de361c7ccfd583 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:10.621063 7420 raft_consensus.cc:2454] T 48528ec242974de4ac20eb0161f0cb05 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:10.621654 6958 leader_election.cc:302] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:10.622009 8495 raft_consensus.cc:2781] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:10.622144 8495 raft_consensus.cc:686] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Becoming Leader. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Running, Role: LEADER
I20240501 20:10:10.622417 8495 consensus_queue.cc:237] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:10.631412 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48528ec242974de4ac20eb0161f0cb05" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:10.631654 7155 raft_consensus.cc:3037] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:10.637492 8877 leader_election.cc:288] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Requested vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:10.634539 7155 raft_consensus.cc:2454] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:10.645287 8878 tablet_bootstrap.cc:492] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:10.645854 8878 tablet_bootstrap.cc:492] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap complete.
I20240501 20:10:10.646186 8878 ts_tablet_manager.cc:1367] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.094s user 0.052s sys 0.006s
I20240501 20:10:10.647307 8878 raft_consensus.cc:348] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:10.647713 8878 raft_consensus.cc:729] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: LEARNER
I20240501 20:10:10.647962 8878 consensus_queue.cc:260] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:10.650506 8878 ts_tablet_manager.cc:1398] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 20:10:10.651455 4519 tablet_copy_service.cc:342] P 3e974c03d3c546f9ab6c91e762a5426b: Request end of tablet copy session b9957d4086bc4a76b18ed9265a5739d5-ffaf0a0105ea4e1f9ed9d90c221ebc8a received from {username='slave'} at 127.29.219.72:58885
I20240501 20:10:10.651607 4519 tablet_copy_service.cc:433] P 3e974c03d3c546f9ab6c91e762a5426b: ending tablet copy session b9957d4086bc4a76b18ed9265a5739d5-ffaf0a0105ea4e1f9ed9d90c221ebc8a on tablet ffaf0a0105ea4e1f9ed9d90c221ebc8a with peer b9957d4086bc4a76b18ed9265a5739d5
I20240501 20:10:10.676728 8925 ts_tablet_manager.cc:902] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d: Initiating tablet copy from peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:10.681178 4087 catalog_manager.cc:5484] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69). New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:10.687592 8882 tablet_bootstrap.cc:492] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:10.695454 8882 ts_tablet_manager.cc:1367] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.108s user 0.011s sys 0.001s
I20240501 20:10:10.696823 8891 tablet_bootstrap.cc:492] T 0dd491a4c3dd42f7910ae61bf68537c3 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:10.697088 8891 ts_tablet_manager.cc:1367] T 0dd491a4c3dd42f7910ae61bf68537c3 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.121s user 0.012s sys 0.000s
I20240501 20:10:10.697901 8891 raft_consensus.cc:348] T 0dd491a4c3dd42f7910ae61bf68537c3 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:10.699448 8882 raft_consensus.cc:348] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:10.699820 8891 raft_consensus.cc:374] T 0dd491a4c3dd42f7910ae61bf68537c3 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:10.699967 8891 raft_consensus.cc:729] T 0dd491a4c3dd42f7910ae61bf68537c3 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:10.700194 8882 raft_consensus.cc:374] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:10.700441 8882 raft_consensus.cc:729] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:10.700670 8891 consensus_queue.cc:260] T 0dd491a4c3dd42f7910ae61bf68537c3 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:10.702172 8883 tablet_bootstrap.cc:492] T 0dd491a4c3dd42f7910ae61bf68537c3 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:10.702338 8883 ts_tablet_manager.cc:1367] T 0dd491a4c3dd42f7910ae61bf68537c3 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.129s user 0.040s sys 0.056s
I20240501 20:10:10.703188 8883 raft_consensus.cc:348] T 0dd491a4c3dd42f7910ae61bf68537c3 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:10.703410 8883 raft_consensus.cc:374] T 0dd491a4c3dd42f7910ae61bf68537c3 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:10.703569 8883 raft_consensus.cc:729] T 0dd491a4c3dd42f7910ae61bf68537c3 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:10.703819 8883 consensus_queue.cc:260] T 0dd491a4c3dd42f7910ae61bf68537c3 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:10.704372 8883 ts_tablet_manager.cc:1398] T 0dd491a4c3dd42f7910ae61bf68537c3 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:10.705801 8891 ts_tablet_manager.cc:1398] T 0dd491a4c3dd42f7910ae61bf68537c3 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 20:10:10.703598 8882 consensus_queue.cc:260] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:10.706650 8882 ts_tablet_manager.cc:1398] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 20:10:10.756106 7420 raft_consensus.cc:1203] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:10.765756 8850 raft_consensus.cc:482] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:10.765935 8850 raft_consensus.cc:504] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:10.766604 8850 leader_election.cc:288] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:10.767622 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd491a4c3dd42f7910ae61bf68537c3" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:10.767890 7022 raft_consensus.cc:2454] T 0dd491a4c3dd42f7910ae61bf68537c3 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:10.768504 7222 leader_election.cc:302] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:10.769071 8853 raft_consensus.cc:2781] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:10.769150 8853 raft_consensus.cc:482] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:10.769229 8853 raft_consensus.cc:3037] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:10.771131 8495 raft_consensus.cc:1050] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d: attempting to promote NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e to VOTER
I20240501 20:10:10.772127 8853 raft_consensus.cc:504] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:10.772161 8495 consensus_queue.cc:237] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } }
I20240501 20:10:10.774785 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd491a4c3dd42f7910ae61bf68537c3" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:10.775067 7420 raft_consensus.cc:2454] T 0dd491a4c3dd42f7910ae61bf68537c3 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:10.777145 8925 tablet_copy_client.cc:308] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.65:42941
I20240501 20:10:10.780769 4254 tablet_copy_service.cc:140] P dca57823003e4729b769ca12367d7530: Received BeginTabletCopySession request for tablet f18af954cefa48d4882313fe18abbd27 from peer ef0a4ada548446b3a531cb39a3c4e38d ({username='slave'} at 127.29.219.69:51289)
I20240501 20:10:10.780941 4254 tablet_copy_service.cc:161] P dca57823003e4729b769ca12367d7530: Beginning new tablet copy session on tablet f18af954cefa48d4882313fe18abbd27 from peer ef0a4ada548446b3a531cb39a3c4e38d at {username='slave'} at 127.29.219.69:51289: session id = ef0a4ada548446b3a531cb39a3c4e38d-f18af954cefa48d4882313fe18abbd27
I20240501 20:10:10.781368 7287 raft_consensus.cc:1261] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:10.781999 8888 consensus_queue.cc:1035] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:10.784101 4254 tablet_copy_source_session.cc:205] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:10.784983 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd491a4c3dd42f7910ae61bf68537c3" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:10.785218 7022 raft_consensus.cc:3037] T 0dd491a4c3dd42f7910ae61bf68537c3 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:10.790958 8842 raft_consensus.cc:482] T ce89f9b962264db7a91685556f1354a8 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:10.791169 8842 raft_consensus.cc:504] T ce89f9b962264db7a91685556f1354a8 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:10.791853 8842 leader_election.cc:288] T ce89f9b962264db7a91685556f1354a8 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:10.797353 4364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce89f9b962264db7a91685556f1354a8" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:10:10.800383 4498 consensus_queue.cc:237] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:10.802354 4631 raft_consensus.cc:1261] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:10.803038 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd491a4c3dd42f7910ae61bf68537c3" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:10.803233 7420 raft_consensus.cc:3037] T 0dd491a4c3dd42f7910ae61bf68537c3 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:10.805523 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce89f9b962264db7a91685556f1354a8" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:10.805809 7020 raft_consensus.cc:2379] T ce89f9b962264db7a91685556f1354a8 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bceeae1c79964acabdae8ff771bf0c97 in current term 1: Already voted for candidate 10b47782558c41ee91f5e164a372049a in this term.
I20240501 20:10:10.805894 7420 raft_consensus.cc:2454] T 0dd491a4c3dd42f7910ae61bf68537c3 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:10.806032 8925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f18af954cefa48d4882313fe18abbd27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:10.807149 8495 consensus_queue.cc:1035] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:10.807948 8853 leader_election.cc:288] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:10.808176 7022 raft_consensus.cc:2454] T 0dd491a4c3dd42f7910ae61bf68537c3 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:10.809258 4364 raft_consensus.cc:1261] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:10.809743 7223 leader_election.cc:302] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:10.810524 8495 consensus_queue.cc:1035] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:10.811105 4233 raft_consensus.cc:1261] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:10.811448 8853 raft_consensus.cc:2781] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:10.811617 8853 raft_consensus.cc:686] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Becoming Leader. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Running, Role: LEADER
I20240501 20:10:10.811937 7222 leader_election.cc:302] T ce89f9b962264db7a91685556f1354a8 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97; no voters: 10b47782558c41ee91f5e164a372049a, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:10.811877 8853 consensus_queue.cc:237] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:10.812570 8850 raft_consensus.cc:3037] T ce89f9b962264db7a91685556f1354a8 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:10.815394 8850 raft_consensus.cc:2726] T ce89f9b962264db7a91685556f1354a8 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:10.816608 8251 consensus_queue.cc:1035] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:10.820184 4627 raft_consensus.cc:1261] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:10.820148 4365 raft_consensus.cc:1203] T d1291a2b3dcc4c55a26df6a5cc390050 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:10.822149 8925 tablet_copy_client.cc:783] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:10.822412 8925 tablet_copy_client.cc:647] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:10.822691 8804 consensus_queue.cc:1035] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:10:10.823451 4435 consensus_peers.cc:485] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b -> Peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): Couldn't send request to peer b9957d4086bc4a76b18ed9265a5739d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f99f5d9d4bf44c39188c8b5d5362b98. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:10.823695 8888 raft_consensus.cc:2932] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:10.830899 7287 raft_consensus.cc:2932] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:10.832224 4091 catalog_manager.cc:5484] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:10.832338 4365 raft_consensus.cc:2932] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:10.834874 4084 catalog_manager.cc:5484] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71). New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:10.838128 4631 raft_consensus.cc:2932] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:10.841614 8804 raft_consensus.cc:2932] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:10.842555 8925 tablet_copy_client.cc:523] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:10.845376 4233 raft_consensus.cc:2932] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:10.846541 8925 tablet_bootstrap.cc:492] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:10.848875 8604 consensus_queue.cc:1035] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:10.852842 4627 raft_consensus.cc:2932] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:10.852944 4088 catalog_manager.cc:5484] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:10.867383 7020 consensus_queue.cc:237] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } }
I20240501 20:10:10.870604 7287 raft_consensus.cc:1261] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:10.871304 8495 consensus_queue.cc:1035] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:10.874504 8495 raft_consensus.cc:2932] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:10.877676 8604 consensus_queue.cc:1035] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:10.877902 4627 raft_consensus.cc:1261] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:10.878634 8884 consensus_queue.cc:1035] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:10.879318 7284 raft_consensus.cc:2932] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:10.880647 4631 raft_consensus.cc:2932] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:10.885217 8374 raft_consensus.cc:1050] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e: attempting to promote NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b to VOTER
I20240501 20:10:10.886185 8374 consensus_queue.cc:237] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:10.890971 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3809cc8ffe6b4a9e99ebbb9d26a65279 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:10.892961 4084 catalog_manager.cc:5484] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 3 to 4, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:10.902009 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet 3809cc8ffe6b4a9e99ebbb9d26a65279 with delete_type TABLET_DATA_TOMBSTONED (TS 10b47782558c41ee91f5e164a372049a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:10.921622 8958 tablet_replica.cc:330] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:10.922111 8958 raft_consensus.cc:2229] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:10.922706 8958 raft_consensus.cc:2258] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:10.923472 8958 ts_tablet_manager.cc:1875] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:10.933022 8958 ts_tablet_manager.cc:1888] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:10.933615 7287 raft_consensus.cc:1261] T 6f2576acd7d34ec1ad27ab04a863df10 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:10.935441 4497 raft_consensus.cc:1261] T 6f2576acd7d34ec1ad27ab04a863df10 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:10.934294 4365 raft_consensus.cc:1261] T 6f2576acd7d34ec1ad27ab04a863df10 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:10.936080 8958 log.cc:1196] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/3809cc8ffe6b4a9e99ebbb9d26a65279
I20240501 20:10:10.936220 8592 consensus_queue.cc:1035] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:10.937415 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet 3809cc8ffe6b4a9e99ebbb9d26a65279 (table rebalancer_extra_table_2 [id=983618faca8342e4aea12a20efab90c0]) successfully deleted
I20240501 20:10:10.936913 8592 consensus_queue.cc:1035] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240501 20:10:10.939390 8314 consensus_queue.cc:1035] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:10.944169 4233 consensus_queue.cc:237] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:10.948246 4498 raft_consensus.cc:1261] T 56cb0964dabd41079d4428787efc9a9d P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:10.948239 4364 raft_consensus.cc:1261] T 56cb0964dabd41079d4428787efc9a9d P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:10.949057 8790 consensus_queue.cc:1035] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:10:10.949543 4167 consensus_peers.cc:485] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 -> Peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Couldn't send request to peer db0742ee3860468590eefef6ae6074fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 56cb0964dabd41079d4428787efc9a9d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:10.949728 8785 consensus_queue.cc:1035] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:10.957255 4084 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:10.961274 8851 consensus_queue.cc:1035] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:10.975502 8785 raft_consensus.cc:2932] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:10.976327 4365 raft_consensus.cc:2932] T 56cb0964dabd41079d4428787efc9a9d P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:10.983750 4084 catalog_manager.cc:5484] T 56cb0964dabd41079d4428787efc9a9d P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:10.991055 7419 raft_consensus.cc:1261] T 0dd491a4c3dd42f7910ae61bf68537c3 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:10.992215 7022 raft_consensus.cc:1261] T 0dd491a4c3dd42f7910ae61bf68537c3 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:10.993450 8853 consensus_queue.cc:1035] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:10.994081 8853 consensus_queue.cc:1035] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:10.995031 8925 tablet_bootstrap.cc:492] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:10.995548 8925 tablet_bootstrap.cc:492] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap complete.
I20240501 20:10:10.995891 8925 ts_tablet_manager.cc:1367] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.149s user 0.060s sys 0.034s
I20240501 20:10:10.997128 8925 raft_consensus.cc:348] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:10.997377 8925 raft_consensus.cc:729] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: LEARNER
I20240501 20:10:10.997583 8925 consensus_queue.cc:260] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:10.998224 8925 ts_tablet_manager.cc:1398] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 20:10:11.001466 4254 tablet_copy_service.cc:342] P dca57823003e4729b769ca12367d7530: Request end of tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-f18af954cefa48d4882313fe18abbd27 received from {username='slave'} at 127.29.219.69:51289
I20240501 20:10:11.001653 4254 tablet_copy_service.cc:433] P dca57823003e4729b769ca12367d7530: ending tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-f18af954cefa48d4882313fe18abbd27 on tablet f18af954cefa48d4882313fe18abbd27 with peer ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:11.007484 8592 raft_consensus.cc:2932] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:11.009044 4496 raft_consensus.cc:2932] T 56cb0964dabd41079d4428787efc9a9d P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:11.015897 4092 catalog_manager.cc:5484] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:11.015355 4497 raft_consensus.cc:2932] T 6f2576acd7d34ec1ad27ab04a863df10 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:11.014693 7285 raft_consensus.cc:2932] T 6f2576acd7d34ec1ad27ab04a863df10 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:11.022583 4084 catalog_manager.cc:5576] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237) does not have the latest schema for tablet 0dd491a4c3dd42f7910ae61bf68537c3 (table rebalancer_dynamic_table [id=7274550d79fa46a59cd847795c41e629]). Expected version 1 got 0
I20240501 20:10:11.023231 8914 consensus_queue.cc:1035] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:11.029583 4631 consensus_queue.cc:237] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:11.041218 8956 mvcc.cc:205] Tried to move back new op lower bound from 7022977888212692992 to 7022977887488131072. Current Snapshot: MvccSnapshot[applied={T|T < 7022977888212692992}]
I20240501 20:10:11.050374 4497 raft_consensus.cc:1261] T 6f2576acd7d34ec1ad27ab04a863df10 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:11.052367 8980 tablet.cc:1722] T 0dd491a4c3dd42f7910ae61bf68537c3 P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:11.053082 8980 tablet.cc:1620] T 0dd491a4c3dd42f7910ae61bf68537c3 P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:11.053664 4498 consensus_queue.cc:237] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:11.053735 8979 tablet.cc:1722] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:11.054188 8314 consensus_queue.cc:1035] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:11.054569 8979 tablet.cc:1620] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:11.057092 8314 raft_consensus.cc:2932] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:11.062012 8884 consensus_queue.cc:1035] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:11.062723 4496 raft_consensus.cc:2932] T 6f2576acd7d34ec1ad27ab04a863df10 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:11.063923 8495 consensus_queue.cc:1035] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:11.066241 4079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6f2576acd7d34ec1ad27ab04a863df10 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:11.068948 4092 catalog_manager.cc:5576] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237) does not have the latest schema for tablet 0dd491a4c3dd42f7910ae61bf68537c3 (table rebalancer_dynamic_table [id=7274550d79fa46a59cd847795c41e629]). Expected version 1 got 0
I20240501 20:10:11.070012 4084 catalog_manager.cc:5484] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:11.077667 4088 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=7274550d79fa46a59cd847795c41e629] alter complete (version 1)
I20240501 20:10:11.080092 7266 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f2576acd7d34ec1ad27ab04a863df10 with delete_type TABLET_DATA_TOMBSTONED (TS bceeae1c79964acabdae8ff771bf0c97 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54166
I20240501 20:10:11.082300 4631 raft_consensus.cc:1261] T 7bcb854a8f7545cc86323abd69c26c23 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:11.082962 8804 consensus_queue.cc:1035] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:10:11.093212 4434 consensus_peers.cc:485] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7bcb854a8f7545cc86323abd69c26c23. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:11.101581 8804 raft_consensus.cc:2932] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:11.107025 4631 raft_consensus.cc:2932] T 7bcb854a8f7545cc86323abd69c26c23 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:11.110688 4366 raft_consensus.cc:1261] T 7bcb854a8f7545cc86323abd69c26c23 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:11.110932 4368 raft_consensus.cc:1261] T 6f2576acd7d34ec1ad27ab04a863df10 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:11.112568 4364 raft_consensus.cc:2932] T 6f2576acd7d34ec1ad27ab04a863df10 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:11.113940 8592 consensus_queue.cc:1035] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:11.117606 8940 consensus_queue.cc:1035] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:11.123404 8986 tablet_replica.cc:330] T 6f2576acd7d34ec1ad27ab04a863df10 P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:11.123621 8986 raft_consensus.cc:2229] T 6f2576acd7d34ec1ad27ab04a863df10 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:11.123901 8986 raft_consensus.cc:2258] T 6f2576acd7d34ec1ad27ab04a863df10 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:11.126144 8986 ts_tablet_manager.cc:1875] T 6f2576acd7d34ec1ad27ab04a863df10 P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:11.132304 8986 ts_tablet_manager.cc:1888] T 6f2576acd7d34ec1ad27ab04a863df10 P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:11.132530 8986 log.cc:1196] T 6f2576acd7d34ec1ad27ab04a863df10 P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/6f2576acd7d34ec1ad27ab04a863df10
I20240501 20:10:11.133492 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet 6f2576acd7d34ec1ad27ab04a863df10 (table rebalancer_extra_table_4 [id=061e2b9b5d53488cbfdc25cfa8acdd76]) successfully deleted
I20240501 20:10:11.135095 4363 raft_consensus.cc:2932] T 6f2576acd7d34ec1ad27ab04a863df10 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:11.143781 8992 tablet.cc:1722] T 0dd491a4c3dd42f7910ae61bf68537c3 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:11.144483 8992 tablet.cc:1620] T 0dd491a4c3dd42f7910ae61bf68537c3 P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:11.147504 4089 catalog_manager.cc:5576] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817) does not have the latest schema for tablet 0dd491a4c3dd42f7910ae61bf68537c3 (table rebalancer_dynamic_table [id=7274550d79fa46a59cd847795c41e629]). Expected version 1 got 0
I20240501 20:10:11.157593 4093 catalog_manager.cc:5484] T 7bcb854a8f7545cc86323abd69c26c23 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:11.158341 8991 ts_tablet_manager.cc:902] T f44675b6874a423398de361c7ccfd583 P 3e974c03d3c546f9ab6c91e762a5426b: Initiating tablet copy from peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:11.160444 4365 raft_consensus.cc:2932] T 7bcb854a8f7545cc86323abd69c26c23 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:11.189834 8801 raft_consensus.cc:1050] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b: attempting to promote NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 to VOTER
I20240501 20:10:11.190845 8801 consensus_queue.cc:237] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:11.203040 8991 tablet_copy_client.cc:308] T f44675b6874a423398de361c7ccfd583 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.65:42941
I20240501 20:10:11.203994 4254 tablet_copy_service.cc:140] P dca57823003e4729b769ca12367d7530: Received BeginTabletCopySession request for tablet f44675b6874a423398de361c7ccfd583 from peer 3e974c03d3c546f9ab6c91e762a5426b ({username='slave'} at 127.29.219.67:51285)
I20240501 20:10:11.204212 4254 tablet_copy_service.cc:161] P dca57823003e4729b769ca12367d7530: Beginning new tablet copy session on tablet f44675b6874a423398de361c7ccfd583 from peer 3e974c03d3c546f9ab6c91e762a5426b at {username='slave'} at 127.29.219.67:51285: session id = 3e974c03d3c546f9ab6c91e762a5426b-f44675b6874a423398de361c7ccfd583
I20240501 20:10:11.206555 7419 raft_consensus.cc:1261] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:11.207509 8804 consensus_queue.cc:1035] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:11.208092 4254 tablet_copy_source_session.cc:205] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:11.209637 8991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f44675b6874a423398de361c7ccfd583. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:11.211290 4365 raft_consensus.cc:1261] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:11.211952 8804 consensus_queue.cc:1035] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:11.216037 4627 raft_consensus.cc:1261] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:11.219643 8940 consensus_queue.cc:1035] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:11.219776 4088 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:11.229045 8940 raft_consensus.cc:2932] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:11.234382 7266 tablet_service.cc:1467] Processing CreateTablet for tablet 965b65ec896648698fbf3e1266d8e678 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=b43133d24f1646bca3476aba5ea45126]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:11.235826 4478 tablet_service.cc:1467] Processing CreateTablet for tablet 965b65ec896648698fbf3e1266d8e678 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=b43133d24f1646bca3476aba5ea45126]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:11.236300 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965b65ec896648698fbf3e1266d8e678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:11.236281 7000 tablet_service.cc:1467] Processing CreateTablet for tablet 965b65ec896648698fbf3e1266d8e678 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=b43133d24f1646bca3476aba5ea45126]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:11.236056 7420 raft_consensus.cc:2932] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:11.236709 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965b65ec896648698fbf3e1266d8e678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:11.239706 7266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965b65ec896648698fbf3e1266d8e678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:11.240483 4087 catalog_manager.cc:5484] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:11.246109 4365 raft_consensus.cc:2932] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:11.260144 4087 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:11.260783 8991 tablet_copy_client.cc:783] T f44675b6874a423398de361c7ccfd583 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:11.261087 8991 tablet_copy_client.cc:647] T f44675b6874a423398de361c7ccfd583 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:11.266589 4498 consensus_queue.cc:237] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:11.268406 8991 tablet_copy_client.cc:523] T f44675b6874a423398de361c7ccfd583 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:11.272459 8991 tablet_bootstrap.cc:492] T f44675b6874a423398de361c7ccfd583 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:11.279075 7419 raft_consensus.cc:1261] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:11.280864 8979 tablet.cc:1722] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:11.281384 8979 tablet.cc:1620] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:11.288544 4365 raft_consensus.cc:1261] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:11.289731 8804 consensus_queue.cc:1035] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:11.291682 8992 tablet.cc:1722] T 0dd491a4c3dd42f7910ae61bf68537c3 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:11.292263 8992 tablet.cc:1620] T 0dd491a4c3dd42f7910ae61bf68537c3 P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:11.293617 8804 raft_consensus.cc:2932] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:11.297647 8251 consensus_queue.cc:1035] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:11.299739 4364 raft_consensus.cc:2932] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:11.301658 7421 raft_consensus.cc:2932] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:11.306867 8980 tablet.cc:1722] T 0dd491a4c3dd42f7910ae61bf68537c3 P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:11.307529 8980 tablet.cc:1620] T 0dd491a4c3dd42f7910ae61bf68537c3 P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:11.318516 4087 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=7274550d79fa46a59cd847795c41e629] alter complete (version 2)
I20240501 20:10:11.320925 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ffaf0a0105ea4e1f9ed9d90c221ebc8a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:11.328440 4610 tablet_service.cc:1467] Processing CreateTablet for tablet ac5256b6282e455dbad777fae1ae8a2b (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=b43133d24f1646bca3476aba5ea45126]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:11.328981 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac5256b6282e455dbad777fae1ae8a2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:11.329555 7135 tablet_service.cc:1467] Processing CreateTablet for tablet ac5256b6282e455dbad777fae1ae8a2b (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=b43133d24f1646bca3476aba5ea45126]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:11.329977 7135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac5256b6282e455dbad777fae1ae8a2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:11.331393 4477 tablet_service.cc:1467] Processing CreateTablet for tablet ac5256b6282e455dbad777fae1ae8a2b (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=b43133d24f1646bca3476aba5ea45126]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:11.331416 7001 tablet_service.cc:1467] Processing CreateTablet for tablet 12f13fda83114ec5b9dcd0fa90ddef7e (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=b43133d24f1646bca3476aba5ea45126]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:11.331833 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac5256b6282e455dbad777fae1ae8a2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:11.331820 7001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12f13fda83114ec5b9dcd0fa90ddef7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:11.334067 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 12f13fda83114ec5b9dcd0fa90ddef7e (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=b43133d24f1646bca3476aba5ea45126]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:11.334527 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12f13fda83114ec5b9dcd0fa90ddef7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:11.336092 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 12f13fda83114ec5b9dcd0fa90ddef7e (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=b43133d24f1646bca3476aba5ea45126]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:11.336520 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12f13fda83114ec5b9dcd0fa90ddef7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:11.340816 8850 raft_consensus.cc:1050] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97: attempting to promote NON_VOTER 10b47782558c41ee91f5e164a372049a to VOTER
I20240501 20:10:11.341905 8850 consensus_queue.cc:237] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } }
I20240501 20:10:11.358081 9004 tablet_bootstrap.cc:492] T 965b65ec896648698fbf3e1266d8e678 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:11.361186 9004 tablet_bootstrap.cc:654] T 965b65ec896648698fbf3e1266d8e678 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:11.365664 9007 tablet_bootstrap.cc:492] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:11.368794 9007 tablet_bootstrap.cc:654] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:11.372624 7420 raft_consensus.cc:1261] T d1291a2b3dcc4c55a26df6a5cc390050 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:11.373919 4363 raft_consensus.cc:1261] T d1291a2b3dcc4c55a26df6a5cc390050 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:11.375602 8853 consensus_queue.cc:1035] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:11.376353 8853 consensus_queue.cc:1035] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:11.377050 9004 tablet_bootstrap.cc:492] T 965b65ec896648698fbf3e1266d8e678 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:10:11.378264 9008 tablet_bootstrap.cc:492] T 12f13fda83114ec5b9dcd0fa90ddef7e P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:11.380442 9008 tablet_bootstrap.cc:654] T 12f13fda83114ec5b9dcd0fa90ddef7e P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:11.380647 9004 ts_tablet_manager.cc:1367] T 965b65ec896648698fbf3e1266d8e678 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.005s
I20240501 20:10:11.383127 9004 raft_consensus.cc:348] T 965b65ec896648698fbf3e1266d8e678 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:11.385046 8842 raft_consensus.cc:2932] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:11.387602 9007 tablet_bootstrap.cc:492] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:11.387585 9014 tablet_bootstrap.cc:492] T ac5256b6282e455dbad777fae1ae8a2b P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:11.388003 9007 ts_tablet_manager.cc:1367] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20240501 20:10:11.389109 9007 raft_consensus.cc:348] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:11.389719 9014 tablet_bootstrap.cc:654] T ac5256b6282e455dbad777fae1ae8a2b P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:11.387768 8991 tablet_bootstrap.cc:492] T f44675b6874a423398de361c7ccfd583 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:11.390426 9004 raft_consensus.cc:374] T 965b65ec896648698fbf3e1266d8e678 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:11.390734 9004 raft_consensus.cc:729] T 965b65ec896648698fbf3e1266d8e678 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:10:11.390812 8991 tablet_bootstrap.cc:492] T f44675b6874a423398de361c7ccfd583 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap complete.
I20240501 20:10:11.391342 8991 ts_tablet_manager.cc:1367] T f44675b6874a423398de361c7ccfd583 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.119s user 0.016s sys 0.052s
I20240501 20:10:11.391793 9004 consensus_queue.cc:260] T 965b65ec896648698fbf3e1266d8e678 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:11.392308 7155 raft_consensus.cc:1261] T d1291a2b3dcc4c55a26df6a5cc390050 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:11.392314 8991 raft_consensus.cc:348] T f44675b6874a423398de361c7ccfd583 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:11.392586 8991 raft_consensus.cc:729] T f44675b6874a423398de361c7ccfd583 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: LEARNER
I20240501 20:10:11.392369 9007 raft_consensus.cc:374] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:11.392865 8991 consensus_queue.cc:260] T f44675b6874a423398de361c7ccfd583 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:11.393136 8853 consensus_queue.cc:1035] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:11.393000 9007 raft_consensus.cc:729] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:11.393538 8991 ts_tablet_manager.cc:1398] T f44675b6874a423398de361c7ccfd583 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:10:11.393267 4363 raft_consensus.cc:2932] T d1291a2b3dcc4c55a26df6a5cc390050 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:11.393994 9004 ts_tablet_manager.cc:1398] T 965b65ec896648698fbf3e1266d8e678 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 20:10:11.394302 9007 consensus_queue.cc:260] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:11.394990 9004 tablet_bootstrap.cc:492] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:11.398284 4254 tablet_copy_service.cc:342] P dca57823003e4729b769ca12367d7530: Request end of tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-f44675b6874a423398de361c7ccfd583 received from {username='slave'} at 127.29.219.67:51285
I20240501 20:10:11.398475 4254 tablet_copy_service.cc:433] P dca57823003e4729b769ca12367d7530: ending tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-f44675b6874a423398de361c7ccfd583 on tablet f44675b6874a423398de361c7ccfd583 with peer 3e974c03d3c546f9ab6c91e762a5426b
I20240501 20:10:11.402477 9004 tablet_bootstrap.cc:654] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:11.402972 9007 ts_tablet_manager.cc:1398] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240501 20:10:11.328155 4088 catalog_manager.cc:5484] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 5 to 6, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:11.404757 9007 tablet_bootstrap.cc:492] T 965b65ec896648698fbf3e1266d8e678 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:11.406705 9011 tablet_bootstrap.cc:492] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:11.408020 4087 catalog_manager.cc:5484] T d1291a2b3dcc4c55a26df6a5cc390050 P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:11.416489 9011 tablet_bootstrap.cc:654] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:11.417531 9008 tablet_bootstrap.cc:492] T 12f13fda83114ec5b9dcd0fa90ddef7e P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:11.418593 9007 tablet_bootstrap.cc:654] T 965b65ec896648698fbf3e1266d8e678 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:11.418857 7155 raft_consensus.cc:2932] T d1291a2b3dcc4c55a26df6a5cc390050 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:11.422811 9008 ts_tablet_manager.cc:1367] T 12f13fda83114ec5b9dcd0fa90ddef7e P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.000s
I20240501 20:10:11.423851 9008 raft_consensus.cc:348] T 12f13fda83114ec5b9dcd0fa90ddef7e P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:11.424146 9008 raft_consensus.cc:374] T 12f13fda83114ec5b9dcd0fa90ddef7e P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:11.424288 9008 raft_consensus.cc:729] T 12f13fda83114ec5b9dcd0fa90ddef7e P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:11.424562 9008 consensus_queue.cc:260] T 12f13fda83114ec5b9dcd0fa90ddef7e P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:11.431609 9008 ts_tablet_manager.cc:1398] T 12f13fda83114ec5b9dcd0fa90ddef7e P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 20:10:11.433789 9004 tablet_bootstrap.cc:492] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:10:11.434139 9004 ts_tablet_manager.cc:1367] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.003s
I20240501 20:10:11.436062 9004 raft_consensus.cc:348] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:11.437307 7285 consensus_queue.cc:237] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } }
I20240501 20:10:11.437394 7419 raft_consensus.cc:2932] T d1291a2b3dcc4c55a26df6a5cc390050 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:11.436507 9004 raft_consensus.cc:374] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:11.439802 9004 raft_consensus.cc:729] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:10:11.440120 9004 consensus_queue.cc:260] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:11.440826 9004 ts_tablet_manager.cc:1398] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240501 20:10:11.441112 4367 raft_consensus.cc:1261] T d1291a2b3dcc4c55a26df6a5cc390050 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:11.441330 9014 tablet_bootstrap.cc:492] T ac5256b6282e455dbad777fae1ae8a2b P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:11.441478 9014 ts_tablet_manager.cc:1367] T ac5256b6282e455dbad777fae1ae8a2b P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.000s
I20240501 20:10:11.441751 8842 consensus_queue.cc:1035] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:11.442289 9014 raft_consensus.cc:348] T ac5256b6282e455dbad777fae1ae8a2b P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:11.442493 9014 raft_consensus.cc:374] T ac5256b6282e455dbad777fae1ae8a2b P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:11.442649 9014 raft_consensus.cc:729] T ac5256b6282e455dbad777fae1ae8a2b P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:11.442961 9014 consensus_queue.cc:260] T ac5256b6282e455dbad777fae1ae8a2b P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:11.447326 9014 ts_tablet_manager.cc:1398] T ac5256b6282e455dbad777fae1ae8a2b P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240501 20:10:11.448185 9007 tablet_bootstrap.cc:492] T 965b65ec896648698fbf3e1266d8e678 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:11.448426 9007 ts_tablet_manager.cc:1367] T 965b65ec896648698fbf3e1266d8e678 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.044s user 0.000s sys 0.011s
I20240501 20:10:11.450748 9007 raft_consensus.cc:348] T 965b65ec896648698fbf3e1266d8e678 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:11.450990 9007 raft_consensus.cc:374] T 965b65ec896648698fbf3e1266d8e678 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:11.451063 9007 raft_consensus.cc:729] T 965b65ec896648698fbf3e1266d8e678 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:11.451309 9007 consensus_queue.cc:260] T 965b65ec896648698fbf3e1266d8e678 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:11.451952 9007 ts_tablet_manager.cc:1398] T 965b65ec896648698fbf3e1266d8e678 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:10:11.455061 4092 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:11.457751 7418 raft_consensus.cc:1261] T d1291a2b3dcc4c55a26df6a5cc390050 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:11.459656 7155 consensus_queue.cc:237] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:11.461244 8850 consensus_queue.cc:1035] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:11.469929 7019 raft_consensus.cc:1261] T 5215e3f2040747f29aa1c9181a628096 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:11.471702 9011 tablet_bootstrap.cc:492] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:11.471897 9011 ts_tablet_manager.cc:1367] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.000s
W20240501 20:10:11.471976 7090 consensus_peers.cc:485] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5215e3f2040747f29aa1c9181a628096. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:11.471868 9020 raft_consensus.cc:2932] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:11.472365 8139 consensus_queue.cc:1035] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:11.472939 9011 raft_consensus.cc:348] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:11.473143 9011 raft_consensus.cc:374] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:11.473238 9011 raft_consensus.cc:729] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:11.473819 9011 consensus_queue.cc:260] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:11.474507 9011 ts_tablet_manager.cc:1398] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:10:11.475026 8925 ts_tablet_manager.cc:902] T 7bcb854a8f7545cc86323abd69c26c23 P ef0a4ada548446b3a531cb39a3c4e38d: Initiating tablet copy from peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:11.475404 7416 raft_consensus.cc:2932] T d1291a2b3dcc4c55a26df6a5cc390050 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:11.477102 8979 tablet.cc:1722] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:11.478458 7850 raft_consensus.cc:2932] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:11.484202 7019 raft_consensus.cc:2932] T 5215e3f2040747f29aa1c9181a628096 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:11.486464 4077 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d1291a2b3dcc4c55a26df6a5cc390050 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:11.488916 7285 raft_consensus.cc:1261] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:11.488905 8992 tablet.cc:1722] T 0dd491a4c3dd42f7910ae61bf68537c3 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:11.495756 8139 consensus_queue.cc:1035] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:11.497733 8980 tablet.cc:1722] T 0dd491a4c3dd42f7910ae61bf68537c3 P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:11.498677 4092 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table_new_name [id=7274550d79fa46a59cd847795c41e629] alter complete (version 3)
I20240501 20:10:11.504081 8925 tablet_copy_client.cc:308] T 7bcb854a8f7545cc86323abd69c26c23 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.67:34595
I20240501 20:10:11.504951 4519 tablet_copy_service.cc:140] P 3e974c03d3c546f9ab6c91e762a5426b: Received BeginTabletCopySession request for tablet 7bcb854a8f7545cc86323abd69c26c23 from peer ef0a4ada548446b3a531cb39a3c4e38d ({username='slave'} at 127.29.219.69:60767)
I20240501 20:10:11.505128 4519 tablet_copy_service.cc:161] P 3e974c03d3c546f9ab6c91e762a5426b: Beginning new tablet copy session on tablet 7bcb854a8f7545cc86323abd69c26c23 from peer ef0a4ada548446b3a531cb39a3c4e38d at {username='slave'} at 127.29.219.69:60767: session id = ef0a4ada548446b3a531cb39a3c4e38d-7bcb854a8f7545cc86323abd69c26c23
I20240501 20:10:11.505970 4092 catalog_manager.cc:5484] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:11.507812 4519 tablet_copy_source_session.cc:205] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:11.509241 8925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bcb854a8f7545cc86323abd69c26c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:11.518576 8925 tablet_copy_client.cc:783] T 7bcb854a8f7545cc86323abd69c26c23 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:11.518914 8925 tablet_copy_client.cc:647] T 7bcb854a8f7545cc86323abd69c26c23 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:11.520942 8925 tablet_copy_client.cc:523] T 7bcb854a8f7545cc86323abd69c26c23 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:11.526232 8925 tablet_bootstrap.cc:492] T 7bcb854a8f7545cc86323abd69c26c23 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:11.546233 4367 raft_consensus.cc:2932] T d1291a2b3dcc4c55a26df6a5cc390050 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:11.553699 9009 tablet_bootstrap.cc:492] T ac5256b6282e455dbad777fae1ae8a2b P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:11.555271 8804 raft_consensus.cc:482] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:11.555128 4087 catalog_manager.cc:5484] T 5215e3f2040747f29aa1c9181a628096 P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:11.555470 8804 raft_consensus.cc:504] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:11.556151 8804 leader_election.cc:288] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:11.556881 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5256b6282e455dbad777fae1ae8a2b" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
W20240501 20:10:11.557653 4435 leader_election.cc:341] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:11.558063 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5256b6282e455dbad777fae1ae8a2b" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:11.558269 7155 raft_consensus.cc:2454] T ac5256b6282e455dbad777fae1ae8a2b P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:10:11.559389 9009 tablet_bootstrap.cc:654] T ac5256b6282e455dbad777fae1ae8a2b P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:11.562105 4432 leader_election.cc:302] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, db0742ee3860468590eefef6ae6074fd; no voters: 1d34e18a1d394d3f91c9f50d1c7bea4e
I20240501 20:10:11.562402 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet ffaf0a0105ea4e1f9ed9d90c221ebc8a with delete_type TABLET_DATA_TOMBSTONED (TS 1d34e18a1d394d3f91c9f50d1c7bea4e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:11.562528 9036 tablet_bootstrap.cc:492] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:11.567955 8940 raft_consensus.cc:2781] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:11.568143 9036 tablet_bootstrap.cc:654] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:11.568131 8940 raft_consensus.cc:482] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:11.568238 8940 raft_consensus.cc:3037] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:11.569476 9025 ts_tablet_manager.cc:902] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5: Initiating tablet copy from peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
W20240501 20:10:11.570642 7089 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.71:39447, user_credentials={real_user=slave}} blocked reactor thread for 81209.5us
I20240501 20:10:11.572260 8940 raft_consensus.cc:504] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:11.572957 8940 leader_election.cc:288] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:11.573333 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5256b6282e455dbad777fae1ae8a2b" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:10:11.573604 9039 tablet_replica.cc:330] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:11.573907 9039 raft_consensus.cc:2229] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:11.574187 9039 pending_rounds.cc:62] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Trying to abort 1 pending ops.
I20240501 20:10:11.574129 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5256b6282e455dbad777fae1ae8a2b" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:11.574354 7153 raft_consensus.cc:3037] T ac5256b6282e455dbad777fae1ae8a2b P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:11.574313 9039 pending_rounds.cc:68] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022977889034948608 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "ffaf0a0105ea4e1f9ed9d90c221ebc8a" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } } }
I20240501 20:10:11.574973 9039 raft_consensus.cc:2894] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 20:10:11.575204 9039 raft_consensus.cc:2866] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:10:11.575919 9039 raft_consensus.cc:2258] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:11.577302 9039 ts_tablet_manager.cc:1875] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:11.578119 7153 raft_consensus.cc:2454] T ac5256b6282e455dbad777fae1ae8a2b P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
W20240501 20:10:11.578678 4435 leader_election.cc:341] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:11.580472 4432 leader_election.cc:302] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, db0742ee3860468590eefef6ae6074fd; no voters: 1d34e18a1d394d3f91c9f50d1c7bea4e
I20240501 20:10:11.580976 8804 raft_consensus.cc:2781] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:11.581126 8804 raft_consensus.cc:686] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:10:11.581357 7135 tablet_service.cc:1514] Processing DeleteTablet for tablet d1291a2b3dcc4c55a26df6a5cc390050 with delete_type TABLET_DATA_TOMBSTONED (TS db0742ee3860468590eefef6ae6074fd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52036
I20240501 20:10:11.581418 8804 consensus_queue.cc:237] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:11.591596 7019 raft_consensus.cc:1203] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:11.609063 9039 ts_tablet_manager.cc:1888] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:11.609244 9039 log.cc:1196] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/ffaf0a0105ea4e1f9ed9d90c221ebc8a
I20240501 20:10:11.611711 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet ffaf0a0105ea4e1f9ed9d90c221ebc8a (table rebalance_test_table_17 [id=09dd73eba68c4f8499a786750d39c1cf]) successfully deleted
I20240501 20:10:11.615787 8888 raft_consensus.cc:482] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:11.616009 8888 raft_consensus.cc:504] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:11.616878 8888 leader_election.cc:288] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:11.618166 4093 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:11.618196 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f13fda83114ec5b9dcd0fa90ddef7e" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:11.618470 4234 raft_consensus.cc:2454] T 12f13fda83114ec5b9dcd0fa90ddef7e P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:11.618587 4093 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:11.619439 6957 leader_election.cc:302] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dca57823003e4729b769ca12367d7530, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:11.619879 8888 raft_consensus.cc:2781] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:11.620010 8888 raft_consensus.cc:482] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:11.620097 8888 raft_consensus.cc:3037] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:11.622139 9042 ts_tablet_manager.cc:902] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd: Initiating tablet copy from peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:11.624092 4092 catalog_manager.cc:5484] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:11.625000 8888 raft_consensus.cc:504] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:11.626325 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f13fda83114ec5b9dcd0fa90ddef7e" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:10:11.626536 4234 raft_consensus.cc:3037] T 12f13fda83114ec5b9dcd0fa90ddef7e P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:11.630151 4234 raft_consensus.cc:2454] T 12f13fda83114ec5b9dcd0fa90ddef7e P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:11.630841 6957 leader_election.cc:302] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dca57823003e4729b769ca12367d7530, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:11.631196 8877 raft_consensus.cc:2781] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:11.631371 8877 raft_consensus.cc:686] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Becoming Leader. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Running, Role: LEADER
I20240501 20:10:11.631748 8877 consensus_queue.cc:237] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:11.634130 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f13fda83114ec5b9dcd0fa90ddef7e" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
W20240501 20:10:11.635583 6958 leader_election.cc:341] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:11.636219 8842 raft_consensus.cc:482] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:11.636384 8842 raft_consensus.cc:504] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:11.636708 8888 leader_election.cc:288] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Requested vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:11.637284 8842 leader_election.cc:288] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:11.637418 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f13fda83114ec5b9dcd0fa90ddef7e" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:11.638065 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965b65ec896648698fbf3e1266d8e678" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:11.638290 4498 raft_consensus.cc:2454] T 965b65ec896648698fbf3e1266d8e678 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
W20240501 20:10:11.638285 6958 leader_election.cc:341] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:11.638967 7222 leader_election.cc:302] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:11.639197 4093 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 0dd491a4c3dd42f7910ae61bf68537c3
I20240501 20:10:11.639737 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965b65ec896648698fbf3e1266d8e678" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:11.639950 7019 raft_consensus.cc:2454] T 965b65ec896648698fbf3e1266d8e678 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:11.640246 7001 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd491a4c3dd42f7910ae61bf68537c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:11 UTC) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:11.640679 8842 raft_consensus.cc:2781] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:11.640803 8842 raft_consensus.cc:482] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:11.640883 8842 raft_consensus.cc:3037] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:11.644129 8842 raft_consensus.cc:504] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:11.645215 4497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965b65ec896648698fbf3e1266d8e678" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:10:11.645473 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "965b65ec896648698fbf3e1266d8e678" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:11.645539 4497 raft_consensus.cc:3037] T 965b65ec896648698fbf3e1266d8e678 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:11.645654 7019 raft_consensus.cc:3037] T 965b65ec896648698fbf3e1266d8e678 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:11.648766 7019 raft_consensus.cc:2454] T 965b65ec896648698fbf3e1266d8e678 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:11.649559 7222 leader_election.cc:302] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:11.649897 9020 raft_consensus.cc:2781] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:11.650063 9020 raft_consensus.cc:686] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Becoming Leader. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Running, Role: LEADER
I20240501 20:10:11.650310 9020 consensus_queue.cc:237] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:11.650905 4497 raft_consensus.cc:2454] T 965b65ec896648698fbf3e1266d8e678 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:11.649897 8842 leader_election.cc:288] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:11.662516 9009 tablet_bootstrap.cc:492] T ac5256b6282e455dbad777fae1ae8a2b P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:11.662882 9009 ts_tablet_manager.cc:1367] T ac5256b6282e455dbad777fae1ae8a2b P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.109s user 0.004s sys 0.000s
I20240501 20:10:11.664264 9009 raft_consensus.cc:348] T ac5256b6282e455dbad777fae1ae8a2b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:11.665108 9036 tablet_bootstrap.cc:492] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:11.665388 9036 ts_tablet_manager.cc:1367] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.103s user 0.004s sys 0.000s
I20240501 20:10:11.666352 9036 raft_consensus.cc:348] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:11.671978 4498 raft_consensus.cc:1203] T f44675b6874a423398de361c7ccfd583 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:11.676985 4092 catalog_manager.cc:5484] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71). New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:11.680594 4234 consensus_queue.cc:237] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:11.681032 7285 raft_consensus.cc:2932] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:11.684479 4092 catalog_manager.cc:5484] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69). New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:11.686910 7266 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd491a4c3dd42f7910ae61bf68537c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:11 UTC) from {username='slave'} at 127.0.0.1:54166
I20240501 20:10:11.691581 9056 tablet_replica.cc:330] T 0dd491a4c3dd42f7910ae61bf68537c3 P ef0a4ada548446b3a531cb39a3c4e38d: stopping tablet replica
I20240501 20:10:11.691948 9056 raft_consensus.cc:2229] T 0dd491a4c3dd42f7910ae61bf68537c3 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:11.692454 9056 raft_consensus.cc:2258] T 0dd491a4c3dd42f7910ae61bf68537c3 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:11.693639 9056 ts_tablet_manager.cc:1875] T 0dd491a4c3dd42f7910ae61bf68537c3 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:11.688091 7401 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd491a4c3dd42f7910ae61bf68537c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:11 UTC) from {username='slave'} at 127.0.0.1:40182
I20240501 20:10:11.700610 9061 tablet_replica.cc:330] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:11.701445 9036 raft_consensus.cc:374] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:11.701900 9036 raft_consensus.cc:729] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:11.702641 9036 consensus_queue.cc:260] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:11.703341 9009 raft_consensus.cc:374] T ac5256b6282e455dbad777fae1ae8a2b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:11.703518 9009 raft_consensus.cc:729] T ac5256b6282e455dbad777fae1ae8a2b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:11.704090 9009 consensus_queue.cc:260] T ac5256b6282e455dbad777fae1ae8a2b P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
W20240501 20:10:11.704890 4169 consensus_peers.cc:485] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53be6ced1df34676b1a34bc5cfe1ac87. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:11.706544 7285 raft_consensus.cc:1261] T 53be6ced1df34676b1a34bc5cfe1ac87 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:11.707022 9061 raft_consensus.cc:2229] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:11.707414 9061 raft_consensus.cc:2258] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:11.711750 9061 ts_tablet_manager.cc:1875] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:11.717010 9036 ts_tablet_manager.cc:1398] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.051s user 0.000s sys 0.002s
I20240501 20:10:11.706190 7420 raft_consensus.cc:1261] T 53be6ced1df34676b1a34bc5cfe1ac87 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:11.717955 8790 consensus_queue.cc:1035] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:11.731376 9009 ts_tablet_manager.cc:1398] T ac5256b6282e455dbad777fae1ae8a2b P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.068s user 0.002s sys 0.000s
I20240501 20:10:11.735184 8925 tablet_bootstrap.cc:492] T 7bcb854a8f7545cc86323abd69c26c23 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:11.735785 8925 tablet_bootstrap.cc:492] T 7bcb854a8f7545cc86323abd69c26c23 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap complete.
I20240501 20:10:11.736084 8925 ts_tablet_manager.cc:1367] T 7bcb854a8f7545cc86323abd69c26c23 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.210s user 0.051s sys 0.020s
I20240501 20:10:11.736918 8925 raft_consensus.cc:348] T 7bcb854a8f7545cc86323abd69c26c23 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:11.737205 8925 raft_consensus.cc:729] T 7bcb854a8f7545cc86323abd69c26c23 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: LEARNER
I20240501 20:10:11.737440 8925 consensus_queue.cc:260] T 7bcb854a8f7545cc86323abd69c26c23 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:11.738077 8925 ts_tablet_manager.cc:1398] T 7bcb854a8f7545cc86323abd69c26c23 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 20:10:11.738888 4519 tablet_copy_service.cc:342] P 3e974c03d3c546f9ab6c91e762a5426b: Request end of tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-7bcb854a8f7545cc86323abd69c26c23 received from {username='slave'} at 127.29.219.69:60767
I20240501 20:10:11.739037 4519 tablet_copy_service.cc:433] P 3e974c03d3c546f9ab6c91e762a5426b: ending tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-7bcb854a8f7545cc86323abd69c26c23 on tablet 7bcb854a8f7545cc86323abd69c26c23 with peer ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:11.745544 9064 tablet_replica.cc:330] T 0dd491a4c3dd42f7910ae61bf68537c3 P b9957d4086bc4a76b18ed9265a5739d5: stopping tablet replica
I20240501 20:10:11.745775 9064 raft_consensus.cc:2229] T 0dd491a4c3dd42f7910ae61bf68537c3 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:11.745999 9064 raft_consensus.cc:2258] T 0dd491a4c3dd42f7910ae61bf68537c3 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:11.750823 9064 ts_tablet_manager.cc:1875] T 0dd491a4c3dd42f7910ae61bf68537c3 P b9957d4086bc4a76b18ed9265a5739d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:11.752470 9025 tablet_copy_client.cc:308] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.67:34595
I20240501 20:10:11.753095 9042 tablet_copy_client.cc:308] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.65:42941
I20240501 20:10:11.753587 4519 tablet_copy_service.cc:140] P 3e974c03d3c546f9ab6c91e762a5426b: Received BeginTabletCopySession request for tablet 4f99f5d9d4bf44c39188c8b5d5362b98 from peer b9957d4086bc4a76b18ed9265a5739d5 ({username='slave'} at 127.29.219.72:58885)
I20240501 20:10:11.753804 4519 tablet_copy_service.cc:161] P 3e974c03d3c546f9ab6c91e762a5426b: Beginning new tablet copy session on tablet 4f99f5d9d4bf44c39188c8b5d5362b98 from peer b9957d4086bc4a76b18ed9265a5739d5 at {username='slave'} at 127.29.219.72:58885: session id = b9957d4086bc4a76b18ed9265a5739d5-4f99f5d9d4bf44c39188c8b5d5362b98
I20240501 20:10:11.753896 4254 tablet_copy_service.cc:140] P dca57823003e4729b769ca12367d7530: Received BeginTabletCopySession request for tablet 56cb0964dabd41079d4428787efc9a9d from peer db0742ee3860468590eefef6ae6074fd ({username='slave'} at 127.29.219.70:47215)
I20240501 20:10:11.754079 4254 tablet_copy_service.cc:161] P dca57823003e4729b769ca12367d7530: Beginning new tablet copy session on tablet 56cb0964dabd41079d4428787efc9a9d from peer db0742ee3860468590eefef6ae6074fd at {username='slave'} at 127.29.219.70:47215: session id = db0742ee3860468590eefef6ae6074fd-56cb0964dabd41079d4428787efc9a9d
I20240501 20:10:11.756981 4519 tablet_copy_source_session.cc:205] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:11.757352 4254 tablet_copy_source_session.cc:205] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:11.760535 8790 consensus_queue.cc:1035] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:11.761898 8790 raft_consensus.cc:2932] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:11.762601 9042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56cb0964dabd41079d4428787efc9a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:11.762665 9044 tablet_replica.cc:330] T d1291a2b3dcc4c55a26df6a5cc390050 P db0742ee3860468590eefef6ae6074fd: stopping tablet replica
I20240501 20:10:11.763046 7419 raft_consensus.cc:2932] T 53be6ced1df34676b1a34bc5cfe1ac87 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:11.771014 8851 raft_consensus.cc:482] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:11.771204 8851 raft_consensus.cc:504] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:11.771889 8851 leader_election.cc:288] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:11.770260 4093 catalog_manager.cc:5484] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:11.771364 9044 raft_consensus.cc:2229] T d1291a2b3dcc4c55a26df6a5cc390050 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:11.772333 9044 raft_consensus.cc:2258] T d1291a2b3dcc4c55a26df6a5cc390050 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:11.771061 9061 ts_tablet_manager.cc:1888] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:11.772591 9061 log.cc:1196] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/0dd491a4c3dd42f7910ae61bf68537c3
I20240501 20:10:11.773140 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f13fda83114ec5b9dcd0fa90ddef7e" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:11.773948 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f13fda83114ec5b9dcd0fa90ddef7e" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:11.774282 4234 raft_consensus.cc:2379] T 12f13fda83114ec5b9dcd0fa90ddef7e P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9957d4086bc4a76b18ed9265a5739d5 in current term 1: Already voted for candidate ef0a4ada548446b3a531cb39a3c4e38d in this term.
I20240501 20:10:11.774917 7355 leader_election.cc:302] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5; no voters: dca57823003e4729b769ca12367d7530, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:11.777422 8851 raft_consensus.cc:3037] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:11.777551 9042 tablet_copy_client.cc:783] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:11.777827 9042 tablet_copy_client.cc:647] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:11.779784 9042 tablet_copy_client.cc:523] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:11.781066 8851 raft_consensus.cc:2726] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:11.782738 9061 ts_tablet_manager.cc:1909] T 0dd491a4c3dd42f7910ae61bf68537c3 P bceeae1c79964acabdae8ff771bf0c97: Deleting consensus metadata
I20240501 20:10:11.783640 9044 ts_tablet_manager.cc:1875] T d1291a2b3dcc4c55a26df6a5cc390050 P db0742ee3860468590eefef6ae6074fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:11.788885 9025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f99f5d9d4bf44c39188c8b5d5362b98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:11.791677 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet 0dd491a4c3dd42f7910ae61bf68537c3 (table rebalancer_dynamic_table_new_name [id=7274550d79fa46a59cd847795c41e629]) successfully deleted
I20240501 20:10:11.792331 4093 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:11.784977 9042 tablet_bootstrap.cc:492] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:11.801519 7135 tablet_service.cc:1467] Processing CreateTablet for tablet 742915bedaad4e4fa7a1a662161bc30d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e4b5dd83375545e28f0bbaaee20a9cb9]), partition=
I20240501 20:10:11.801926 7135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 742915bedaad4e4fa7a1a662161bc30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:11.801959 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 742915bedaad4e4fa7a1a662161bc30d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e4b5dd83375545e28f0bbaaee20a9cb9]), partition=
I20240501 20:10:11.802362 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 742915bedaad4e4fa7a1a662161bc30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:11.803203 4347 tablet_service.cc:1467] Processing CreateTablet for tablet 742915bedaad4e4fa7a1a662161bc30d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e4b5dd83375545e28f0bbaaee20a9cb9]), partition=
I20240501 20:10:11.803545 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 742915bedaad4e4fa7a1a662161bc30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:11.809154 9014 tablet_bootstrap.cc:492] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:11.810858 9014 tablet_bootstrap.cc:654] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:11.813231 9056 ts_tablet_manager.cc:1888] T 0dd491a4c3dd42f7910ae61bf68537c3 P ef0a4ada548446b3a531cb39a3c4e38d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:11.813584 9056 log.cc:1196] T 0dd491a4c3dd42f7910ae61bf68537c3 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/wals/0dd491a4c3dd42f7910ae61bf68537c3
I20240501 20:10:11.814123 9056 ts_tablet_manager.cc:1909] T 0dd491a4c3dd42f7910ae61bf68537c3 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting consensus metadata
I20240501 20:10:11.818708 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet 0dd491a4c3dd42f7910ae61bf68537c3 (table rebalancer_dynamic_table_new_name [id=7274550d79fa46a59cd847795c41e629]) successfully deleted
I20240501 20:10:11.820158 9036 tablet_bootstrap.cc:492] T 742915bedaad4e4fa7a1a662161bc30d P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:11.822011 9036 tablet_bootstrap.cc:654] T 742915bedaad4e4fa7a1a662161bc30d P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:11.834273 9064 ts_tablet_manager.cc:1888] T 0dd491a4c3dd42f7910ae61bf68537c3 P b9957d4086bc4a76b18ed9265a5739d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:11.834913 9064 log.cc:1196] T 0dd491a4c3dd42f7910ae61bf68537c3 P b9957d4086bc4a76b18ed9265a5739d5: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/wals/0dd491a4c3dd42f7910ae61bf68537c3
I20240501 20:10:11.836218 7284 raft_consensus.cc:2932] T 53be6ced1df34676b1a34bc5cfe1ac87 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:11.838238 9064 ts_tablet_manager.cc:1909] T 0dd491a4c3dd42f7910ae61bf68537c3 P b9957d4086bc4a76b18ed9265a5739d5: Deleting consensus metadata
I20240501 20:10:11.843544 9070 tablet_bootstrap.cc:492] T 742915bedaad4e4fa7a1a662161bc30d P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:11.844170 9044 ts_tablet_manager.cc:1888] T d1291a2b3dcc4c55a26df6a5cc390050 P db0742ee3860468590eefef6ae6074fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:11.844591 9025 tablet_copy_client.cc:783] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:11.845029 9025 tablet_copy_client.cc:647] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:11.845032 9044 log.cc:1196] T d1291a2b3dcc4c55a26df6a5cc390050 P db0742ee3860468590eefef6ae6074fd: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/wals/d1291a2b3dcc4c55a26df6a5cc390050
I20240501 20:10:11.845310 4079 catalog_manager.cc:4830] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): tablet 0dd491a4c3dd42f7910ae61bf68537c3 (table rebalancer_dynamic_table_new_name [id=7274550d79fa46a59cd847795c41e629]) successfully deleted
I20240501 20:10:11.846004 9014 tablet_bootstrap.cc:492] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:11.846004 9036 tablet_bootstrap.cc:492] T 742915bedaad4e4fa7a1a662161bc30d P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:11.846185 9036 ts_tablet_manager.cc:1367] T 742915bedaad4e4fa7a1a662161bc30d P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.008s
I20240501 20:10:11.846244 9014 ts_tablet_manager.cc:1367] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.007s
I20240501 20:10:11.847147 9036 raft_consensus.cc:348] T 742915bedaad4e4fa7a1a662161bc30d P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:11.847186 9014 raft_consensus.cc:348] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:11.847462 9014 raft_consensus.cc:374] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:11.847589 9014 raft_consensus.cc:729] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:11.847746 9036 raft_consensus.cc:374] T 742915bedaad4e4fa7a1a662161bc30d P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:11.847842 9036 raft_consensus.cc:729] T 742915bedaad4e4fa7a1a662161bc30d P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:11.848513 9014 consensus_queue.cc:260] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:11.849709 9036 consensus_queue.cc:260] T 742915bedaad4e4fa7a1a662161bc30d P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:11.852058 9036 ts_tablet_manager.cc:1398] T 742915bedaad4e4fa7a1a662161bc30d P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240501 20:10:11.852964 9014 ts_tablet_manager.cc:1398] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 20:10:11.855731 9070 tablet_bootstrap.cc:654] T 742915bedaad4e4fa7a1a662161bc30d P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:11.857079 9025 tablet_copy_client.cc:523] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:11.863350 4076 catalog_manager.cc:4830] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): tablet d1291a2b3dcc4c55a26df6a5cc390050 (table rebalancer_extra_table_0 [id=40f3268f46a14344bdf5de27989a6f1a]) successfully deleted
I20240501 20:10:11.882403 9025 tablet_bootstrap.cc:492] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:11.891700 9077 ts_tablet_manager.cc:902] T 5215e3f2040747f29aa1c9181a628096 P dca57823003e4729b769ca12367d7530: Initiating tablet copy from peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:11.900971 9070 tablet_bootstrap.cc:492] T 742915bedaad4e4fa7a1a662161bc30d P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:10:11.901311 9070 ts_tablet_manager.cc:1367] T 742915bedaad4e4fa7a1a662161bc30d P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.058s user 0.011s sys 0.008s
I20240501 20:10:11.902297 9070 raft_consensus.cc:348] T 742915bedaad4e4fa7a1a662161bc30d P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:11.902653 9070 raft_consensus.cc:374] T 742915bedaad4e4fa7a1a662161bc30d P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:11.902805 9070 raft_consensus.cc:729] T 742915bedaad4e4fa7a1a662161bc30d P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:10:11.903126 9070 consensus_queue.cc:260] T 742915bedaad4e4fa7a1a662161bc30d P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:11.904800 9070 ts_tablet_manager.cc:1398] T 742915bedaad4e4fa7a1a662161bc30d P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:10:11.909440 9032 raft_consensus.cc:482] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:11.909628 9032 raft_consensus.cc:504] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:11.910600 9032 leader_election.cc:288] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:11.911756 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "742915bedaad4e4fa7a1a662161bc30d" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:11.912020 7420 raft_consensus.cc:2454] T 742915bedaad4e4fa7a1a662161bc30d P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:11.912470 7091 leader_election.cc:302] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:11.914752 9032 raft_consensus.cc:2781] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:11.914908 9032 raft_consensus.cc:482] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:11.915001 9032 raft_consensus.cc:3037] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:11.917896 9032 raft_consensus.cc:504] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:11.920506 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "742915bedaad4e4fa7a1a662161bc30d" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:11.920748 7419 raft_consensus.cc:3037] T 742915bedaad4e4fa7a1a662161bc30d P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:11.923974 7419 raft_consensus.cc:2454] T 742915bedaad4e4fa7a1a662161bc30d P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:11.924552 7091 leader_election.cc:302] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:11.925132 8139 raft_consensus.cc:2781] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:11.925282 8139 raft_consensus.cc:686] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Becoming Leader. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Running, Role: LEADER
I20240501 20:10:11.925586 8139 consensus_queue.cc:237] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:11.928977 9032 leader_election.cc:288] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Requested vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:11.946573 9077 tablet_copy_client.cc:308] T 5215e3f2040747f29aa1c9181a628096 P dca57823003e4729b769ca12367d7530: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.70:36321
I20240501 20:10:11.946965 8314 raft_consensus.cc:482] T ac5256b6282e455dbad777fae1ae8a2b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:11.947139 8314 raft_consensus.cc:504] T ac5256b6282e455dbad777fae1ae8a2b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:11.947818 8314 leader_election.cc:288] T ac5256b6282e455dbad777fae1ae8a2b P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:11.948508 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5256b6282e455dbad777fae1ae8a2b" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:11.948691 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac5256b6282e455dbad777fae1ae8a2b" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:11.948943 7155 raft_consensus.cc:2379] T ac5256b6282e455dbad777fae1ae8a2b P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in current term 1: Already voted for candidate 3e974c03d3c546f9ab6c91e762a5426b in this term.
I20240501 20:10:11.949932 4562 leader_election.cc:302] T ac5256b6282e455dbad777fae1ae8a2b P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e; no voters: 3e974c03d3c546f9ab6c91e762a5426b, db0742ee3860468590eefef6ae6074fd
I20240501 20:10:11.950594 8314 raft_consensus.cc:3037] T ac5256b6282e455dbad777fae1ae8a2b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:11.953465 8314 raft_consensus.cc:2726] T ac5256b6282e455dbad777fae1ae8a2b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:11.954502 7175 tablet_copy_service.cc:140] P db0742ee3860468590eefef6ae6074fd: Received BeginTabletCopySession request for tablet 5215e3f2040747f29aa1c9181a628096 from peer dca57823003e4729b769ca12367d7530 ({username='slave'} at 127.29.219.65:43859)
I20240501 20:10:11.954699 7175 tablet_copy_service.cc:161] P db0742ee3860468590eefef6ae6074fd: Beginning new tablet copy session on tablet 5215e3f2040747f29aa1c9181a628096 from peer dca57823003e4729b769ca12367d7530 at {username='slave'} at 127.29.219.65:43859: session id = dca57823003e4729b769ca12367d7530-5215e3f2040747f29aa1c9181a628096
I20240501 20:10:11.957907 7175 tablet_copy_source_session.cc:205] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:11.960084 9077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5215e3f2040747f29aa1c9181a628096. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:11.965150 4093 catalog_manager.cc:5484] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd reported cstate change: term changed from 0 to 1, leader changed from <none> to db0742ee3860468590eefef6ae6074fd (127.29.219.70). New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:11.971654 9077 tablet_copy_client.cc:783] T 5215e3f2040747f29aa1c9181a628096 P dca57823003e4729b769ca12367d7530: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:11.972095 9077 tablet_copy_client.cc:647] T 5215e3f2040747f29aa1c9181a628096 P dca57823003e4729b769ca12367d7530: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:11.974092 9077 tablet_copy_client.cc:523] T 5215e3f2040747f29aa1c9181a628096 P dca57823003e4729b769ca12367d7530: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:11.975845 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "742915bedaad4e4fa7a1a662161bc30d" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:10:11.976089 4365 raft_consensus.cc:2454] T 742915bedaad4e4fa7a1a662161bc30d P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:11.976490 4367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "742915bedaad4e4fa7a1a662161bc30d" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:10:11.976649 4367 raft_consensus.cc:3037] T 742915bedaad4e4fa7a1a662161bc30d P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:11.978466 9077 tablet_bootstrap.cc:492] T 5215e3f2040747f29aa1c9181a628096 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:11.979674 4367 raft_consensus.cc:2454] T 742915bedaad4e4fa7a1a662161bc30d P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:11.999068 9077 tablet_bootstrap.cc:492] T 5215e3f2040747f29aa1c9181a628096 P dca57823003e4729b769ca12367d7530: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:11.999724 9077 tablet_bootstrap.cc:492] T 5215e3f2040747f29aa1c9181a628096 P dca57823003e4729b769ca12367d7530: Bootstrap complete.
I20240501 20:10:12.000087 9077 ts_tablet_manager.cc:1367] T 5215e3f2040747f29aa1c9181a628096 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.005s
I20240501 20:10:12.001313 9077 raft_consensus.cc:348] T 5215e3f2040747f29aa1c9181a628096 P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:12.001698 9077 raft_consensus.cc:729] T 5215e3f2040747f29aa1c9181a628096 P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: LEARNER
I20240501 20:10:12.002048 9077 consensus_queue.cc:260] T 5215e3f2040747f29aa1c9181a628096 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:12.006739 9077 ts_tablet_manager.cc:1398] T 5215e3f2040747f29aa1c9181a628096 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.007s user 0.003s sys 0.001s
I20240501 20:10:12.008208 7175 tablet_copy_service.cc:342] P db0742ee3860468590eefef6ae6074fd: Request end of tablet copy session dca57823003e4729b769ca12367d7530-5215e3f2040747f29aa1c9181a628096 received from {username='slave'} at 127.29.219.65:43859
I20240501 20:10:12.008375 7175 tablet_copy_service.cc:433] P db0742ee3860468590eefef6ae6074fd: ending tablet copy session dca57823003e4729b769ca12367d7530-5215e3f2040747f29aa1c9181a628096 on tablet 5215e3f2040747f29aa1c9181a628096 with peer dca57823003e4729b769ca12367d7530
I20240501 20:10:12.017431 8804 consensus_queue.cc:1035] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:12.044234 8940 consensus_queue.cc:1035] T ac5256b6282e455dbad777fae1ae8a2b P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:12.047508 9042 tablet_bootstrap.cc:492] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:12.048070 9042 tablet_bootstrap.cc:492] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd: Bootstrap complete.
I20240501 20:10:12.048378 9042 ts_tablet_manager.cc:1367] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.263s user 0.053s sys 0.004s
I20240501 20:10:12.049235 9042 raft_consensus.cc:348] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:12.053169 9042 raft_consensus.cc:729] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: LEARNER
I20240501 20:10:12.053471 9042 consensus_queue.cc:260] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:12.054240 9042 ts_tablet_manager.cc:1398] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240501 20:10:12.055056 4254 tablet_copy_service.cc:342] P dca57823003e4729b769ca12367d7530: Request end of tablet copy session db0742ee3860468590eefef6ae6074fd-56cb0964dabd41079d4428787efc9a9d received from {username='slave'} at 127.29.219.70:47215
I20240501 20:10:12.055212 4254 tablet_copy_service.cc:433] P dca57823003e4729b769ca12367d7530: ending tablet copy session db0742ee3860468590eefef6ae6074fd-56cb0964dabd41079d4428787efc9a9d on tablet 56cb0964dabd41079d4428787efc9a9d with peer db0742ee3860468590eefef6ae6074fd
I20240501 20:10:12.061031 9025 tablet_bootstrap.cc:492] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:12.061439 9025 tablet_bootstrap.cc:492] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap complete.
I20240501 20:10:12.061713 9025 ts_tablet_manager.cc:1367] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.179s user 0.046s sys 0.003s
I20240501 20:10:12.062449 9025 raft_consensus.cc:348] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:12.062690 9025 raft_consensus.cc:729] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: LEARNER
I20240501 20:10:12.062909 9025 consensus_queue.cc:260] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:12.063516 9025 ts_tablet_manager.cc:1398] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 20:10:12.065985 4519 tablet_copy_service.cc:342] P 3e974c03d3c546f9ab6c91e762a5426b: Request end of tablet copy session b9957d4086bc4a76b18ed9265a5739d5-4f99f5d9d4bf44c39188c8b5d5362b98 received from {username='slave'} at 127.29.219.72:58885
I20240501 20:10:12.066107 4519 tablet_copy_service.cc:433] P 3e974c03d3c546f9ab6c91e762a5426b: ending tablet copy session b9957d4086bc4a76b18ed9265a5739d5-4f99f5d9d4bf44c39188c8b5d5362b98 on tablet 4f99f5d9d4bf44c39188c8b5d5362b98 with peer b9957d4086bc4a76b18ed9265a5739d5
I20240501 20:10:12.069985 7019 raft_consensus.cc:1203] T 7bcb854a8f7545cc86323abd69c26c23 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:12.084779 4093 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:12.090808 4367 raft_consensus.cc:1261] T 742915bedaad4e4fa7a1a662161bc30d P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:12.090922 7419 raft_consensus.cc:1261] T 742915bedaad4e4fa7a1a662161bc30d P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:12.091522 9032 consensus_queue.cc:1035] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:12.095405 9032 consensus_queue.cc:1035] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:12.110913 9065 raft_consensus.cc:1050] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530: attempting to promote NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d to VOTER
I20240501 20:10:12.112196 9065 consensus_queue.cc:237] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:12.117847 7019 raft_consensus.cc:1261] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:12.118711 8785 consensus_queue.cc:1035] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:12.123773 4627 raft_consensus.cc:1261] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:12.124651 8790 consensus_queue.cc:1035] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:12.125010 4093 catalog_manager.cc:5576] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689) does not have the latest schema for tablet 742915bedaad4e4fa7a1a662161bc30d (table rebalancer_dynamic_table [id=e4b5dd83375545e28f0bbaaee20a9cb9]). Expected version 1 got 0
I20240501 20:10:12.128821 4498 raft_consensus.cc:1261] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:12.130573 8790 consensus_queue.cc:1035] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:12.135551 8785 raft_consensus.cc:2932] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:12.140857 4498 raft_consensus.cc:2932] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:12.146418 9020 consensus_queue.cc:1035] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:12.162268 4093 catalog_manager.cc:5576] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817) does not have the latest schema for tablet 742915bedaad4e4fa7a1a662161bc30d (table rebalancer_dynamic_table [id=e4b5dd83375545e28f0bbaaee20a9cb9]). Expected version 1 got 0
I20240501 20:10:12.171520 4091 catalog_manager.cc:5576] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689) does not have the latest schema for tablet 742915bedaad4e4fa7a1a662161bc30d (table rebalancer_dynamic_table [id=e4b5dd83375545e28f0bbaaee20a9cb9]). Expected version 1 got 0
I20240501 20:10:12.173674 9117 tablet.cc:1722] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:12.174238 9118 tablet.cc:1722] T 742915bedaad4e4fa7a1a662161bc30d P 10b47782558c41ee91f5e164a372049a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:12.175518 9117 tablet.cc:1620] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:10:12.176445 9121 tablet.cc:1722] T 742915bedaad4e4fa7a1a662161bc30d P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:12.178267 4091 catalog_manager.cc:5576] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817) does not have the latest schema for tablet 742915bedaad4e4fa7a1a662161bc30d (table rebalancer_dynamic_table [id=e4b5dd83375545e28f0bbaaee20a9cb9]). Expected version 1 got 0
I20240501 20:10:12.180119 9118 tablet.cc:1620] T 742915bedaad4e4fa7a1a662161bc30d P 10b47782558c41ee91f5e164a372049a: MemRowSet was empty: no flush needed.
I20240501 20:10:12.182345 9121 tablet.cc:1620] T 742915bedaad4e4fa7a1a662161bc30d P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:12.194379 4087 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:12.195672 4091 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=e4b5dd83375545e28f0bbaaee20a9cb9] alter complete (version 1)
I20240501 20:10:12.200078 8877 consensus_queue.cc:1035] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:12.208938 9020 consensus_queue.cc:1035] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:12.211211 7266 tablet_service.cc:1467] Processing CreateTablet for tablet 7afb8e3154a64be78b8041b137b3681a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=b446a76fc1184ee88b0b983584d3f5f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:12.211934 7266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7afb8e3154a64be78b8041b137b3681a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:12.212536 7001 tablet_service.cc:1467] Processing CreateTablet for tablet 7afb8e3154a64be78b8041b137b3681a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=b446a76fc1184ee88b0b983584d3f5f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:12.212684 7000 tablet_service.cc:1467] Processing CreateTablet for tablet 030335599b754c81810fe0d139ef51f1 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=b446a76fc1184ee88b0b983584d3f5f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:12.212991 7001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7afb8e3154a64be78b8041b137b3681a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:12.213268 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 030335599b754c81810fe0d139ef51f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:12.213958 7135 tablet_service.cc:1467] Processing CreateTablet for tablet 030335599b754c81810fe0d139ef51f1 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=b446a76fc1184ee88b0b983584d3f5f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:12.214383 7135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 030335599b754c81810fe0d139ef51f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:12.215059 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 7afb8e3154a64be78b8041b137b3681a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=b446a76fc1184ee88b0b983584d3f5f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:12.215487 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7afb8e3154a64be78b8041b137b3681a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:12.215806 7002 tablet_service.cc:1467] Processing CreateTablet for tablet e7f35eafb0be4f4eab73133917b22bbf (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=b446a76fc1184ee88b0b983584d3f5f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:12.216183 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7f35eafb0be4f4eab73133917b22bbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:12.224193 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 030335599b754c81810fe0d139ef51f1 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=b446a76fc1184ee88b0b983584d3f5f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:12.224720 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 030335599b754c81810fe0d139ef51f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:12.226871 8877 consensus_queue.cc:1035] T 12f13fda83114ec5b9dcd0fa90ddef7e P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240501 20:10:12.231791 4609 tablet_service.cc:1467] Processing CreateTablet for tablet e7f35eafb0be4f4eab73133917b22bbf (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=b446a76fc1184ee88b0b983584d3f5f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:12.232367 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7f35eafb0be4f4eab73133917b22bbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:12.232955 9125 tablet_bootstrap.cc:492] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:12.234097 9036 tablet_bootstrap.cc:492] T 030335599b754c81810fe0d139ef51f1 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:12.235316 9125 tablet_bootstrap.cc:654] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:12.235649 9036 tablet_bootstrap.cc:654] T 030335599b754c81810fe0d139ef51f1 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:12.236621 7267 tablet_service.cc:1467] Processing CreateTablet for tablet e7f35eafb0be4f4eab73133917b22bbf (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=b446a76fc1184ee88b0b983584d3f5f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:12.237109 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7f35eafb0be4f4eab73133917b22bbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:12.241773 9014 tablet_bootstrap.cc:492] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:12.244200 9014 tablet_bootstrap.cc:654] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:12.250154 9014 tablet_bootstrap.cc:492] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:12.250313 9014 ts_tablet_manager.cc:1367] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240501 20:10:12.251120 9014 raft_consensus.cc:348] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:12.251384 9014 raft_consensus.cc:374] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:12.251518 9014 raft_consensus.cc:729] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:12.251780 9014 consensus_queue.cc:260] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:12.252328 9014 ts_tablet_manager.cc:1398] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:12.256999 9128 tablet_bootstrap.cc:492] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:12.260687 9128 tablet_bootstrap.cc:654] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:12.261870 9125 tablet_bootstrap.cc:492] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:12.262137 9125 ts_tablet_manager.cc:1367] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.006s
I20240501 20:10:12.263550 9125 raft_consensus.cc:348] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:12.263901 9125 raft_consensus.cc:374] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:12.264107 9125 raft_consensus.cc:729] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:12.265571 9125 consensus_queue.cc:260] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:12.266469 9125 ts_tablet_manager.cc:1398] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 20:10:12.267180 9125 tablet_bootstrap.cc:492] T 030335599b754c81810fe0d139ef51f1 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:12.270982 9125 tablet_bootstrap.cc:654] T 030335599b754c81810fe0d139ef51f1 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:12.272941 9128 tablet_bootstrap.cc:492] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:12.273742 9128 ts_tablet_manager.cc:1367] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20240501 20:10:12.276566 9128 raft_consensus.cc:348] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:12.277822 9128 raft_consensus.cc:374] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:12.278354 9128 raft_consensus.cc:729] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:12.283800 9125 tablet_bootstrap.cc:492] T 030335599b754c81810fe0d139ef51f1 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:12.280583 9128 consensus_queue.cc:260] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:12.288125 9128 ts_tablet_manager.cc:1398] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240501 20:10:12.289767 9125 ts_tablet_manager.cc:1367] T 030335599b754c81810fe0d139ef51f1 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.001s
I20240501 20:10:12.291177 9065 raft_consensus.cc:1050] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530: attempting to promote NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b to VOTER
I20240501 20:10:12.292145 9125 raft_consensus.cc:348] T 030335599b754c81810fe0d139ef51f1 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:12.292222 9065 consensus_queue.cc:237] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:12.292838 9125 raft_consensus.cc:374] T 030335599b754c81810fe0d139ef51f1 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:12.293110 9125 raft_consensus.cc:729] T 030335599b754c81810fe0d139ef51f1 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:12.293540 9125 consensus_queue.cc:260] T 030335599b754c81810fe0d139ef51f1 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:12.294747 9125 ts_tablet_manager.cc:1398] T 030335599b754c81810fe0d139ef51f1 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:10:12.295518 9125 tablet_bootstrap.cc:492] T e7f35eafb0be4f4eab73133917b22bbf P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:12.298395 9125 tablet_bootstrap.cc:654] T e7f35eafb0be4f4eab73133917b22bbf P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:12.300073 7284 raft_consensus.cc:1261] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:12.302136 9065 consensus_queue.cc:1035] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:12.304672 4365 raft_consensus.cc:1261] T f44675b6874a423398de361c7ccfd583 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:12.305447 9065 consensus_queue.cc:1035] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:12.310504 9125 tablet_bootstrap.cc:492] T e7f35eafb0be4f4eab73133917b22bbf P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:12.311924 9125 ts_tablet_manager.cc:1367] T e7f35eafb0be4f4eab73133917b22bbf P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20240501 20:10:12.315423 9125 raft_consensus.cc:348] T e7f35eafb0be4f4eab73133917b22bbf P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:12.316174 9125 raft_consensus.cc:374] T e7f35eafb0be4f4eab73133917b22bbf P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:12.316449 9125 raft_consensus.cc:729] T e7f35eafb0be4f4eab73133917b22bbf P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:12.317124 9125 consensus_queue.cc:260] T e7f35eafb0be4f4eab73133917b22bbf P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:12.318603 9125 ts_tablet_manager.cc:1398] T e7f35eafb0be4f4eab73133917b22bbf P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240501 20:10:12.322135 9036 tablet_bootstrap.cc:492] T 030335599b754c81810fe0d139ef51f1 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:12.322350 9036 ts_tablet_manager.cc:1367] T 030335599b754c81810fe0d139ef51f1 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.088s user 0.016s sys 0.025s
I20240501 20:10:12.323721 9036 raft_consensus.cc:348] T 030335599b754c81810fe0d139ef51f1 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:12.324124 9036 raft_consensus.cc:374] T 030335599b754c81810fe0d139ef51f1 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:12.324276 9036 raft_consensus.cc:729] T 030335599b754c81810fe0d139ef51f1 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:12.324658 9036 consensus_queue.cc:260] T 030335599b754c81810fe0d139ef51f1 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:12.325644 9036 ts_tablet_manager.cc:1398] T 030335599b754c81810fe0d139ef51f1 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 20:10:12.335156 4233 raft_consensus.cc:1203] T 5215e3f2040747f29aa1c9181a628096 P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:12.336061 9131 raft_consensus.cc:482] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:12.336223 9131 raft_consensus.cc:504] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:12.336916 9131 leader_election.cc:288] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
W20240501 20:10:12.338238 7223 leader_election.cc:341] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Not found: Tablet not found: e7f35eafb0be4f4eab73133917b22bbf
I20240501 20:10:12.338389 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7f35eafb0be4f4eab73133917b22bbf" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:12.338743 7020 raft_consensus.cc:2454] T e7f35eafb0be4f4eab73133917b22bbf P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:12.341019 7222 leader_election.cc:302] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d; no voters: 1d34e18a1d394d3f91c9f50d1c7bea4e
I20240501 20:10:12.341531 9131 raft_consensus.cc:2781] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:12.341768 9131 raft_consensus.cc:482] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:12.341905 9131 raft_consensus.cc:3037] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:12.349305 9131 raft_consensus.cc:504] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:12.350598 9131 leader_election.cc:288] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
W20240501 20:10:12.351780 7223 leader_election.cc:341] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Not found: Tablet not found: e7f35eafb0be4f4eab73133917b22bbf
I20240501 20:10:12.352569 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7f35eafb0be4f4eab73133917b22bbf" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:12.352875 7020 raft_consensus.cc:3037] T e7f35eafb0be4f4eab73133917b22bbf P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:12.356539 4627 raft_consensus.cc:2932] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:12.360615 7020 raft_consensus.cc:2454] T e7f35eafb0be4f4eab73133917b22bbf P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:12.361363 7222 leader_election.cc:302] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d; no voters: 1d34e18a1d394d3f91c9f50d1c7bea4e
I20240501 20:10:12.369295 4091 catalog_manager.cc:5484] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:12.370977 9128 tablet_bootstrap.cc:492] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:12.372931 9128 tablet_bootstrap.cc:654] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:12.379041 9131 raft_consensus.cc:2781] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:12.380522 9131 raft_consensus.cc:686] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Becoming Leader. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Running, Role: LEADER
I20240501 20:10:12.380995 9131 consensus_queue.cc:237] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:12.383512 4091 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:12.398234 4499 raft_consensus.cc:1261] T f44675b6874a423398de361c7ccfd583 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:12.399642 9065 consensus_queue.cc:1035] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:12.410574 9121 tablet.cc:1722] T 742915bedaad4e4fa7a1a662161bc30d P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:12.411172 9121 tablet.cc:1620] T 742915bedaad4e4fa7a1a662161bc30d P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:12.420145 9134 ts_tablet_manager.cc:902] T 53be6ced1df34676b1a34bc5cfe1ac87 P 3e974c03d3c546f9ab6c91e762a5426b: Initiating tablet copy from peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:12.422992 9117 tablet.cc:1722] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:12.423503 9117 tablet.cc:1620] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:10:12.427907 9118 tablet.cc:1722] T 742915bedaad4e4fa7a1a662161bc30d P 10b47782558c41ee91f5e164a372049a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:12.428457 9118 tablet.cc:1620] T 742915bedaad4e4fa7a1a662161bc30d P 10b47782558c41ee91f5e164a372049a: MemRowSet was empty: no flush needed.
I20240501 20:10:12.437750 9139 tablet_bootstrap.cc:492] T e7f35eafb0be4f4eab73133917b22bbf P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:12.440264 9139 tablet_bootstrap.cc:654] T e7f35eafb0be4f4eab73133917b22bbf P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:12.456916 8801 raft_consensus.cc:1050] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b: attempting to promote NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d to VOTER
I20240501 20:10:12.458114 8801 consensus_queue.cc:237] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:12.463327 4092 catalog_manager.cc:5484] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71). New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:12.465147 7020 raft_consensus.cc:1261] T 7bcb854a8f7545cc86323abd69c26c23 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:12.465854 8804 consensus_queue.cc:1035] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:12.467454 4367 raft_consensus.cc:1261] T 7bcb854a8f7545cc86323abd69c26c23 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:12.468079 8804 consensus_queue.cc:1035] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:12.457859 4091 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=e4b5dd83375545e28f0bbaaee20a9cb9] alter complete (version 2)
I20240501 20:10:12.490021 9065 raft_consensus.cc:2932] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:12.500185 9033 raft_consensus.cc:482] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:12.500378 9033 raft_consensus.cc:504] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:12.501210 9033 leader_election.cc:288] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:12.500963 4499 raft_consensus.cc:2932] T f44675b6874a423398de361c7ccfd583 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:12.502913 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "030335599b754c81810fe0d139ef51f1" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:12.505342 7419 raft_consensus.cc:2454] T 030335599b754c81810fe0d139ef51f1 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:12.506029 7091 leader_election.cc:302] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:12.505575 4367 raft_consensus.cc:2932] T 7bcb854a8f7545cc86323abd69c26c23 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:12.506657 9033 raft_consensus.cc:2781] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:12.506811 9033 raft_consensus.cc:482] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:12.506896 9033 raft_consensus.cc:3037] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:12.510481 9033 raft_consensus.cc:504] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:12.512833 7018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "030335599b754c81810fe0d139ef51f1" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:12.513131 7018 raft_consensus.cc:3037] T 030335599b754c81810fe0d139ef51f1 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:12.514256 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "030335599b754c81810fe0d139ef51f1" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:12.514451 7419 raft_consensus.cc:3037] T 030335599b754c81810fe0d139ef51f1 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:12.515162 4091 catalog_manager.cc:5484] T 7bcb854a8f7545cc86323abd69c26c23 P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:12.516255 7018 raft_consensus.cc:2454] T 030335599b754c81810fe0d139ef51f1 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:12.516723 7090 leader_election.cc:302] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db0742ee3860468590eefef6ae6074fd, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:12.517066 9032 raft_consensus.cc:2781] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:12.517218 9032 raft_consensus.cc:686] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Becoming Leader. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Running, Role: LEADER
I20240501 20:10:12.517542 9032 consensus_queue.cc:237] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:12.527261 7421 raft_consensus.cc:1203] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:12.530959 9033 leader_election.cc:288] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Requested vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:12.540516 7021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "030335599b754c81810fe0d139ef51f1" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:12.542284 7021 raft_consensus.cc:2362] T 030335599b754c81810fe0d139ef51f1 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1. Re-sending same reply.
I20240501 20:10:12.542883 7419 raft_consensus.cc:2454] T 030335599b754c81810fe0d139ef51f1 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:12.504105 8804 raft_consensus.cc:2932] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:12.556210 7155 raft_consensus.cc:1203] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 20:10:12.562019 9137 log.cc:924] Time spent T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97: Append to log took a long time: real 0.120s user 0.000s sys 0.000s
I20240501 20:10:12.530397 9134 tablet_copy_client.cc:308] T 53be6ced1df34676b1a34bc5cfe1ac87 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.65:42941
I20240501 20:10:12.564006 4254 tablet_copy_service.cc:140] P dca57823003e4729b769ca12367d7530: Received BeginTabletCopySession request for tablet 53be6ced1df34676b1a34bc5cfe1ac87 from peer 3e974c03d3c546f9ab6c91e762a5426b ({username='slave'} at 127.29.219.67:51285)
I20240501 20:10:12.564217 4254 tablet_copy_service.cc:161] P dca57823003e4729b769ca12367d7530: Beginning new tablet copy session on tablet 53be6ced1df34676b1a34bc5cfe1ac87 from peer 3e974c03d3c546f9ab6c91e762a5426b at {username='slave'} at 127.29.219.67:51285: session id = 3e974c03d3c546f9ab6c91e762a5426b-53be6ced1df34676b1a34bc5cfe1ac87
I20240501 20:10:12.567975 4254 tablet_copy_source_session.cc:205] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:12.561477 4365 raft_consensus.cc:2932] T f44675b6874a423398de361c7ccfd583 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:12.554438 7021 raft_consensus.cc:2932] T 7bcb854a8f7545cc86323abd69c26c23 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:12.574409 7284 raft_consensus.cc:2932] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:12.574990 4497 consensus_queue.cc:237] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:12.576215 9128 tablet_bootstrap.cc:492] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:12.577502 9134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53be6ced1df34676b1a34bc5cfe1ac87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:12.583024 9128 ts_tablet_manager.cc:1367] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.212s user 0.021s sys 0.055s
I20240501 20:10:12.584777 9128 raft_consensus.cc:348] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:12.591601 9128 raft_consensus.cc:374] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:12.591943 9128 raft_consensus.cc:729] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:12.592382 9128 consensus_queue.cc:260] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:12.588918 7020 raft_consensus.cc:2932] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:12.593204 9128 ts_tablet_manager.cc:1398] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240501 20:10:12.577363 4363 raft_consensus.cc:1261] T 7bcb854a8f7545cc86323abd69c26c23 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:12.605392 8804 consensus_queue.cc:1035] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:12.608603 8804 raft_consensus.cc:2932] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:12.611614 4367 raft_consensus.cc:2932] T 7bcb854a8f7545cc86323abd69c26c23 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:12.620312 4091 catalog_manager.cc:5484] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:12.620606 4088 catalog_manager.cc:5484] T 7bcb854a8f7545cc86323abd69c26c23 P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index 5 to 6, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:12.624084 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7bcb854a8f7545cc86323abd69c26c23 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:12.629091 4093 catalog_manager.cc:5484] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd reported cstate change: term changed from 0 to 1, leader changed from <none> to db0742ee3860468590eefef6ae6074fd (127.29.219.70). New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:12.631109 4234 consensus_queue.cc:237] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:12.633560 4631 raft_consensus.cc:1261] T 7bcb854a8f7545cc86323abd69c26c23 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:12.683017 4091 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:12.687992 7022 raft_consensus.cc:1261] T 7bcb854a8f7545cc86323abd69c26c23 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:12.689217 8804 consensus_queue.cc:1035] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:12.691840 7019 raft_consensus.cc:2932] T 7bcb854a8f7545cc86323abd69c26c23 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:12.693284 9117 tablet.cc:1722] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:12.698386 9134 tablet_copy_client.cc:783] T 53be6ced1df34676b1a34bc5cfe1ac87 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:12.699584 9118 tablet.cc:1722] T 742915bedaad4e4fa7a1a662161bc30d P 10b47782558c41ee91f5e164a372049a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:12.705858 9121 tablet.cc:1722] T 742915bedaad4e4fa7a1a662161bc30d P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:12.707599 9134 tablet_copy_client.cc:647] T 53be6ced1df34676b1a34bc5cfe1ac87 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:12.710709 9134 tablet_copy_client.cc:523] T 53be6ced1df34676b1a34bc5cfe1ac87 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:12.712437 9159 raft_consensus.cc:482] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:12.712740 9159 raft_consensus.cc:504] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:12.718691 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afb8e3154a64be78b8041b137b3681a" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:12.718982 7284 raft_consensus.cc:2454] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:12.719599 6956 leader_election.cc:302] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:12.720292 9159 leader_election.cc:288] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:12.720535 9159 raft_consensus.cc:2781] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:12.720690 9159 raft_consensus.cc:482] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:12.720845 9159 raft_consensus.cc:3037] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:12.726792 9159 raft_consensus.cc:504] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:12.727089 4499 raft_consensus.cc:1261] T f44675b6874a423398de361c7ccfd583 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:12.728049 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afb8e3154a64be78b8041b137b3681a" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:12.728283 7284 raft_consensus.cc:3037] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:12.730541 9122 consensus_queue.cc:1035] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:12.731739 7284 raft_consensus.cc:2454] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:12.732251 6956 leader_election.cc:302] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:12.732722 8877 raft_consensus.cc:2781] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:12.732986 8877 raft_consensus.cc:686] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Becoming Leader. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Running, Role: LEADER
I20240501 20:10:12.734144 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afb8e3154a64be78b8041b137b3681a" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:12.734416 4087 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table_new_name [id=e4b5dd83375545e28f0bbaaee20a9cb9] alter complete (version 3)
I20240501 20:10:12.734611 4365 raft_consensus.cc:1261] T f44675b6874a423398de361c7ccfd583 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:12.735320 9122 consensus_queue.cc:1035] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:12.737927 9159 leader_election.cc:288] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Requested vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
W20240501 20:10:12.740640 6958 leader_election.cc:341] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:12.740314 8785 raft_consensus.cc:2932] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:12.741431 8877 consensus_queue.cc:237] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:12.743141 9139 tablet_bootstrap.cc:492] T e7f35eafb0be4f4eab73133917b22bbf P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:12.743333 9139 ts_tablet_manager.cc:1367] T e7f35eafb0be4f4eab73133917b22bbf P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.306s user 0.005s sys 0.005s
I20240501 20:10:12.744141 9139 raft_consensus.cc:348] T e7f35eafb0be4f4eab73133917b22bbf P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:12.744447 9139 raft_consensus.cc:374] T e7f35eafb0be4f4eab73133917b22bbf P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:12.744576 9139 raft_consensus.cc:729] T e7f35eafb0be4f4eab73133917b22bbf P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:12.744891 9139 consensus_queue.cc:260] T e7f35eafb0be4f4eab73133917b22bbf P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:12.747135 9139 ts_tablet_manager.cc:1398] T e7f35eafb0be4f4eab73133917b22bbf P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:10:12.747571 9139 tablet_bootstrap.cc:492] T 7afb8e3154a64be78b8041b137b3681a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:12.749449 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f44675b6874a423398de361c7ccfd583 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:12.749361 4367 raft_consensus.cc:2932] T f44675b6874a423398de361c7ccfd583 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:12.751852 4092 catalog_manager.cc:5484] T f44675b6874a423398de361c7ccfd583 P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:12.756778 7266 tablet_service.cc:1514] Processing DeleteTablet for tablet f44675b6874a423398de361c7ccfd583 with delete_type TABLET_DATA_TOMBSTONED (TS bceeae1c79964acabdae8ff771bf0c97 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54166
I20240501 20:10:12.759305 4497 raft_consensus.cc:2932] T f44675b6874a423398de361c7ccfd583 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:12.761519 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afb8e3154a64be78b8041b137b3681a" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:10:12.764734 4609 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bcb854a8f7545cc86323abd69c26c23 with delete_type TABLET_DATA_TOMBSTONED (TS 1d34e18a1d394d3f91c9f50d1c7bea4e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51236
W20240501 20:10:12.777315 6958 leader_election.cc:341] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:12.792579 9134 tablet_bootstrap.cc:492] T 53be6ced1df34676b1a34bc5cfe1ac87 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:12.803053 4087 catalog_manager.cc:5484] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69). New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:12.814172 9139 tablet_bootstrap.cc:654] T 7afb8e3154a64be78b8041b137b3681a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
W20240501 20:10:12.813618 4435 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.68:45205, user_credentials={real_user=slave}} blocked reactor thread for 48919.8us
I20240501 20:10:12.818994 9131 consensus_queue.cc:1035] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:12.821475 9168 tablet_replica.cc:330] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:12.822019 9170 tablet_replica.cc:330] T 7bcb854a8f7545cc86323abd69c26c23 P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:12.828102 9170 raft_consensus.cc:2229] T 7bcb854a8f7545cc86323abd69c26c23 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:12.828760 9170 raft_consensus.cc:2258] T 7bcb854a8f7545cc86323abd69c26c23 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:12.829926 9170 ts_tablet_manager.cc:1875] T 7bcb854a8f7545cc86323abd69c26c23 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:12.838335 4087 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:12.839834 4087 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:12.845113 4087 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 742915bedaad4e4fa7a1a662161bc30d
I20240501 20:10:12.846092 7135 tablet_service.cc:1514] Processing DeleteTablet for tablet 742915bedaad4e4fa7a1a662161bc30d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:12 UTC) from {username='slave'} at 127.0.0.1:52036
I20240501 20:10:12.847303 9178 tablet_replica.cc:330] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd: stopping tablet replica
I20240501 20:10:12.848178 7401 tablet_service.cc:1514] Processing DeleteTablet for tablet 742915bedaad4e4fa7a1a662161bc30d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:12 UTC) from {username='slave'} at 127.0.0.1:40182
I20240501 20:10:12.848429 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet 742915bedaad4e4fa7a1a662161bc30d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:12 UTC) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:12.849506 9168 raft_consensus.cc:2229] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:12.849987 9168 raft_consensus.cc:2258] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:12.855381 9178 raft_consensus.cc:2229] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:12.856218 9178 raft_consensus.cc:2258] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:12.858284 9178 ts_tablet_manager.cc:1875] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:12.859375 9139 tablet_bootstrap.cc:492] T 7afb8e3154a64be78b8041b137b3681a P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:12.859444 9168 ts_tablet_manager.cc:1875] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:12.859566 9139 ts_tablet_manager.cc:1367] T 7afb8e3154a64be78b8041b137b3681a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.112s user 0.004s sys 0.000s
I20240501 20:10:12.860452 9139 raft_consensus.cc:348] T 7afb8e3154a64be78b8041b137b3681a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:12.860746 9139 raft_consensus.cc:374] T 7afb8e3154a64be78b8041b137b3681a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:12.860847 9139 raft_consensus.cc:729] T 7afb8e3154a64be78b8041b137b3681a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:12.861140 9139 consensus_queue.cc:260] T 7afb8e3154a64be78b8041b137b3681a P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:12.861812 9139 ts_tablet_manager.cc:1398] T 7afb8e3154a64be78b8041b137b3681a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:12.865621 9180 tablet_replica.cc:330] T 742915bedaad4e4fa7a1a662161bc30d P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:12.866107 9180 raft_consensus.cc:2229] T 742915bedaad4e4fa7a1a662161bc30d P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:12.866468 9180 raft_consensus.cc:2258] T 742915bedaad4e4fa7a1a662161bc30d P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:12.867030 9179 tablet_replica.cc:330] T 742915bedaad4e4fa7a1a662161bc30d P b9957d4086bc4a76b18ed9265a5739d5: stopping tablet replica
I20240501 20:10:12.869722 9178 ts_tablet_manager.cc:1888] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:12.869789 9179 raft_consensus.cc:2229] T 742915bedaad4e4fa7a1a662161bc30d P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:12.869948 9178 log.cc:1196] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/wals/742915bedaad4e4fa7a1a662161bc30d
I20240501 20:10:12.870117 9179 raft_consensus.cc:2258] T 742915bedaad4e4fa7a1a662161bc30d P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:12.868219 9180 ts_tablet_manager.cc:1875] T 742915bedaad4e4fa7a1a662161bc30d P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:12.870507 9178 ts_tablet_manager.cc:1909] T 742915bedaad4e4fa7a1a662161bc30d P db0742ee3860468590eefef6ae6074fd: Deleting consensus metadata
I20240501 20:10:12.871794 4076 catalog_manager.cc:4830] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): tablet 742915bedaad4e4fa7a1a662161bc30d (table rebalancer_dynamic_table_new_name [id=e4b5dd83375545e28f0bbaaee20a9cb9]) successfully deleted
I20240501 20:10:12.873875 9168 ts_tablet_manager.cc:1888] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:12.875725 9179 ts_tablet_manager.cc:1875] T 742915bedaad4e4fa7a1a662161bc30d P b9957d4086bc4a76b18ed9265a5739d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:12.876299 9134 tablet_bootstrap.cc:492] T 53be6ced1df34676b1a34bc5cfe1ac87 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:12.876840 9134 tablet_bootstrap.cc:492] T 53be6ced1df34676b1a34bc5cfe1ac87 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap complete.
I20240501 20:10:12.877180 9134 ts_tablet_manager.cc:1367] T 53be6ced1df34676b1a34bc5cfe1ac87 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.085s user 0.008s sys 0.001s
I20240501 20:10:12.878110 9134 raft_consensus.cc:348] T 53be6ced1df34676b1a34bc5cfe1ac87 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:12.879951 9134 raft_consensus.cc:729] T 53be6ced1df34676b1a34bc5cfe1ac87 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: LEARNER
I20240501 20:10:12.889748 4627 raft_consensus.cc:3037] T e7f35eafb0be4f4eab73133917b22bbf P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:12.893139 9168 log.cc:1196] T f44675b6874a423398de361c7ccfd583 P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/f44675b6874a423398de361c7ccfd583
I20240501 20:10:12.895116 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet f44675b6874a423398de361c7ccfd583 (table rebalancer_extra_table_8 [id=c0709b4853a54a8ea0cd4c5dfb289ba1]) successfully deleted
I20240501 20:10:12.895638 9170 ts_tablet_manager.cc:1888] T 7bcb854a8f7545cc86323abd69c26c23 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 20:10:12.896082 9134 consensus_queue.cc:260] T 53be6ced1df34676b1a34bc5cfe1ac87 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:12.897516 9180 ts_tablet_manager.cc:1888] T 742915bedaad4e4fa7a1a662161bc30d P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:12.897733 9180 log.cc:1196] T 742915bedaad4e4fa7a1a662161bc30d P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/742915bedaad4e4fa7a1a662161bc30d
I20240501 20:10:12.898928 9180 ts_tablet_manager.cc:1909] T 742915bedaad4e4fa7a1a662161bc30d P 10b47782558c41ee91f5e164a372049a: Deleting consensus metadata
I20240501 20:10:12.899086 9134 ts_tablet_manager.cc:1398] T 53be6ced1df34676b1a34bc5cfe1ac87 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240501 20:10:12.899544 9131 consensus_queue.cc:1035] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240501 20:10:12.899758 4254 tablet_copy_service.cc:342] P dca57823003e4729b769ca12367d7530: Request end of tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-53be6ced1df34676b1a34bc5cfe1ac87 received from {username='slave'} at 127.29.219.67:51285
I20240501 20:10:12.899899 4254 tablet_copy_service.cc:433] P dca57823003e4729b769ca12367d7530: ending tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-53be6ced1df34676b1a34bc5cfe1ac87 on tablet 53be6ced1df34676b1a34bc5cfe1ac87 with peer 3e974c03d3c546f9ab6c91e762a5426b
I20240501 20:10:12.900511 9179 ts_tablet_manager.cc:1888] T 742915bedaad4e4fa7a1a662161bc30d P b9957d4086bc4a76b18ed9265a5739d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:12.901451 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet 742915bedaad4e4fa7a1a662161bc30d (table rebalancer_dynamic_table_new_name [id=e4b5dd83375545e28f0bbaaee20a9cb9]) successfully deleted
I20240501 20:10:12.901906 9179 log.cc:1196] T 742915bedaad4e4fa7a1a662161bc30d P b9957d4086bc4a76b18ed9265a5739d5: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/wals/742915bedaad4e4fa7a1a662161bc30d
I20240501 20:10:12.902535 9179 ts_tablet_manager.cc:1909] T 742915bedaad4e4fa7a1a662161bc30d P b9957d4086bc4a76b18ed9265a5739d5: Deleting consensus metadata
I20240501 20:10:12.895781 9170 log.cc:1196] T 7bcb854a8f7545cc86323abd69c26c23 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/7bcb854a8f7545cc86323abd69c26c23
I20240501 20:10:12.904209 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet 7bcb854a8f7545cc86323abd69c26c23 (table rebalance_test_table_19 [id=8de1e355c457486da8bba4c628d52c17]) successfully deleted
I20240501 20:10:12.907279 4079 catalog_manager.cc:4830] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): tablet 742915bedaad4e4fa7a1a662161bc30d (table rebalancer_dynamic_table_new_name [id=e4b5dd83375545e28f0bbaaee20a9cb9]) successfully deleted
I20240501 20:10:12.950932 4087 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:12.953624 9033 raft_consensus.cc:1050] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd: attempting to promote NON_VOTER dca57823003e4729b769ca12367d7530 to VOTER
I20240501 20:10:12.954558 9033 consensus_queue.cc:237] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:12.958357 4234 raft_consensus.cc:1261] T 5215e3f2040747f29aa1c9181a628096 P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:12.959085 9163 raft_consensus.cc:482] T 7afb8e3154a64be78b8041b137b3681a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:12.959178 8139 consensus_queue.cc:1035] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:12.959306 9163 raft_consensus.cc:504] T 7afb8e3154a64be78b8041b137b3681a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:12.960397 9163 leader_election.cc:288] T 7afb8e3154a64be78b8041b137b3681a P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:12.961922 7285 raft_consensus.cc:1261] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:12.961926 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afb8e3154a64be78b8041b137b3681a" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:12.962579 7022 raft_consensus.cc:1261] T 5215e3f2040747f29aa1c9181a628096 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:12.962739 9033 consensus_queue.cc:1035] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:12.963109 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afb8e3154a64be78b8041b137b3681a" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:12.963411 7284 raft_consensus.cc:2379] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in current term 1: Already voted for candidate ef0a4ada548446b3a531cb39a3c4e38d in this term.
I20240501 20:10:12.964254 4564 leader_election.cc:302] T 7afb8e3154a64be78b8041b137b3681a P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e; no voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:12.965020 9163 raft_consensus.cc:3037] T 7afb8e3154a64be78b8041b137b3681a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:12.965024 8139 consensus_queue.cc:1035] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:12.968323 9163 raft_consensus.cc:2726] T 7afb8e3154a64be78b8041b137b3681a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:12.976225 8139 consensus_queue.cc:1035] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:12.982276 8139 consensus_queue.cc:1035] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:12.985114 7266 tablet_service.cc:1467] Processing CreateTablet for tablet 21f156318bd945dca004ae1afcc40a2c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ce784d0bfe4b43798e9861c3fa5866c2]), partition=
I20240501 20:10:12.985054 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 21f156318bd945dca004ae1afcc40a2c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ce784d0bfe4b43798e9861c3fa5866c2]), partition=
I20240501 20:10:12.985499 7266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21f156318bd945dca004ae1afcc40a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:12.985558 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21f156318bd945dca004ae1afcc40a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:12.991262 7135 tablet_service.cc:1467] Processing CreateTablet for tablet 21f156318bd945dca004ae1afcc40a2c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ce784d0bfe4b43798e9861c3fa5866c2]), partition=
I20240501 20:10:12.991685 7135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21f156318bd945dca004ae1afcc40a2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:13.001804 4233 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f18af954cefa48d4882313fe18abbd27"
dest_uuid: "dca57823003e4729b769ca12367d7530"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59714
I20240501 20:10:13.002104 4233 raft_consensus.cc:595] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:10:13.003643 8139 raft_consensus.cc:2932] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:13.018471 7285 raft_consensus.cc:2932] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:13.023945 4093 catalog_manager.cc:5484] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:13.029964 9128 tablet_bootstrap.cc:492] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:13.032338 9128 tablet_bootstrap.cc:654] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:13.056034 7022 raft_consensus.cc:2932] T 5215e3f2040747f29aa1c9181a628096 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:13.058727 9065 raft_consensus.cc:1050] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530: attempting to promote NON_VOTER db0742ee3860468590eefef6ae6074fd to VOTER
I20240501 20:10:13.059641 9065 consensus_queue.cc:237] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:13.061012 9206 tablet_bootstrap.cc:492] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:13.066371 9128 tablet_bootstrap.cc:492] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:13.066568 9128 ts_tablet_manager.cc:1367] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.005s
I20240501 20:10:13.067873 9198 tablet_bootstrap.cc:492] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:13.068601 9128 raft_consensus.cc:348] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:13.068869 9128 raft_consensus.cc:374] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:13.069026 9128 raft_consensus.cc:729] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:13.069388 9128 consensus_queue.cc:260] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:13.071316 9128 ts_tablet_manager.cc:1398] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:10:13.071259 7153 consensus_queue.cc:237] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:13.072911 9198 tablet_bootstrap.cc:654] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:13.073524 4233 raft_consensus.cc:1261] T 5215e3f2040747f29aa1c9181a628096 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:13.073573 7285 raft_consensus.cc:1261] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:13.074286 9032 consensus_queue.cc:1035] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:13.078330 9032 consensus_queue.cc:1035] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:13.079339 8139 raft_consensus.cc:2932] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:13.080180 7285 raft_consensus.cc:2932] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:13.082026 4233 raft_consensus.cc:2932] T 5215e3f2040747f29aa1c9181a628096 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:13.084887 4076 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5215e3f2040747f29aa1c9181a628096 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:13.088163 4088 catalog_manager.cc:5484] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:13.095788 9206 tablet_bootstrap.cc:654] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:13.097651 4365 raft_consensus.cc:1261] T 56cb0964dabd41079d4428787efc9a9d P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:13.098662 4499 raft_consensus.cc:1261] T 56cb0964dabd41079d4428787efc9a9d P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:13.095798 7153 raft_consensus.cc:1261] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:13.100168 4234 raft_consensus.cc:2932] T 5215e3f2040747f29aa1c9181a628096 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:13.100574 9122 consensus_queue.cc:1035] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:13.101238 9122 consensus_queue.cc:1035] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:13.102156 7002 tablet_service.cc:1514] Processing DeleteTablet for tablet 5215e3f2040747f29aa1c9181a628096 with delete_type TABLET_DATA_TOMBSTONED (TS ef0a4ada548446b3a531cb39a3c4e38d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:13.102989 9122 consensus_queue.cc:1035] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:13.113281 9198 tablet_bootstrap.cc:492] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:13.114753 9198 ts_tablet_manager.cc:1367] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.047s user 0.013s sys 0.016s
I20240501 20:10:13.117190 9198 raft_consensus.cc:348] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:13.117934 9198 raft_consensus.cc:374] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:13.118187 9198 raft_consensus.cc:729] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:13.118603 9198 consensus_queue.cc:260] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:13.122541 9198 ts_tablet_manager.cc:1398] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240501 20:10:13.131819 9169 raft_consensus.cc:1050] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b: attempting to promote NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 to VOTER
I20240501 20:10:13.132977 9169 consensus_queue.cc:237] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:13.136927 9206 tablet_bootstrap.cc:492] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:13.138330 9206 ts_tablet_manager.cc:1367] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.077s user 0.017s sys 0.017s
I20240501 20:10:13.139516 8785 raft_consensus.cc:979] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530: : Instructing follower ef0a4ada548446b3a531cb39a3c4e38d to start an election
I20240501 20:10:13.139669 8785 raft_consensus.cc:1067] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Signalling peer ef0a4ada548446b3a531cb39a3c4e38d to start an election
I20240501 20:10:13.140084 9206 raft_consensus.cc:348] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:13.141850 9206 raft_consensus.cc:374] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:13.142292 9206 raft_consensus.cc:729] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:13.143052 9206 consensus_queue.cc:260] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:13.144608 9206 ts_tablet_manager.cc:1398] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:10:13.145444 7022 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f18af954cefa48d4882313fe18abbd27"
dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
from {username='slave'} at 127.29.219.65:40207
I20240501 20:10:13.145733 7022 raft_consensus.cc:482] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:10:13.146823 7022 raft_consensus.cc:3037] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:13.188055 7022 raft_consensus.cc:504] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:13.189134 7022 leader_election.cc:288] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 2 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:13.189997 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f18af954cefa48d4882313fe18abbd27" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:10:13.190282 4234 raft_consensus.cc:3032] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:10:13.190407 4234 raft_consensus.cc:729] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:10:13.190834 4234 consensus_queue.cc:260] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:13.191394 4234 raft_consensus.cc:3037] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:13.195901 4496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f18af954cefa48d4882313fe18abbd27" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:10:13.196204 4496 raft_consensus.cc:3037] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:13.207188 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f18af954cefa48d4882313fe18abbd27" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:10:13.207497 4630 raft_consensus.cc:3037] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:13.214236 9213 tablet_replica.cc:330] T 5215e3f2040747f29aa1c9181a628096 P ef0a4ada548446b3a531cb39a3c4e38d: stopping tablet replica
I20240501 20:10:13.215438 4630 raft_consensus.cc:2454] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 2.
I20240501 20:10:13.216862 4233 raft_consensus.cc:1261] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:13.217120 9213 raft_consensus.cc:2229] T 5215e3f2040747f29aa1c9181a628096 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:13.217537 9213 raft_consensus.cc:2258] T 5215e3f2040747f29aa1c9181a628096 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:13.217481 8790 raft_consensus.cc:2932] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:13.217999 9169 consensus_queue.cc:1035] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:13.218600 9213 ts_tablet_manager.cc:1875] T 5215e3f2040747f29aa1c9181a628096 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:13.220726 7155 raft_consensus.cc:2932] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:13.226078 9213 ts_tablet_manager.cc:1888] T 5215e3f2040747f29aa1c9181a628096 P ef0a4ada548446b3a531cb39a3c4e38d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:13.226279 9213 log.cc:1196] T 5215e3f2040747f29aa1c9181a628096 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/wals/5215e3f2040747f29aa1c9181a628096
I20240501 20:10:13.227356 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet 5215e3f2040747f29aa1c9181a628096 (table rebalancer_extra_table_0 [id=40f3268f46a14344bdf5de27989a6f1a]) successfully deleted
I20240501 20:10:13.228899 4087 catalog_manager.cc:5484] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:13.233004 4496 raft_consensus.cc:2454] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 2.
I20240501 20:10:13.233776 4365 raft_consensus.cc:2932] T 56cb0964dabd41079d4428787efc9a9d P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:13.234582 6957 leader_election.cc:302] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:13.235517 4627 raft_consensus.cc:1261] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:13.236423 9158 raft_consensus.cc:2781] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:13.236594 9158 raft_consensus.cc:686] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d [term 2 LEADER]: Becoming Leader. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Running, Role: LEADER
I20240501 20:10:13.236969 9158 consensus_queue.cc:237] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:13.238193 9131 raft_consensus.cc:482] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:13.238399 9131 raft_consensus.cc:504] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:13.238711 4234 raft_consensus.cc:2454] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 2.
I20240501 20:10:13.239254 9131 leader_election.cc:288] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:13.240514 4496 raft_consensus.cc:2932] T 56cb0964dabd41079d4428787efc9a9d P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:13.251955 4233 consensus_queue.cc:237] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:13.257272 4496 raft_consensus.cc:1261] T 56cb0964dabd41079d4428787efc9a9d P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:13.257566 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21f156318bd945dca004ae1afcc40a2c" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:13.257783 7153 raft_consensus.cc:1261] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:13.258708 9169 consensus_queue.cc:1035] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:13.259009 7420 raft_consensus.cc:1261] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:13.257812 7155 raft_consensus.cc:2454] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:13.259927 9169 consensus_queue.cc:1035] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:13.260656 8785 consensus_queue.cc:1035] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:13.261121 7220 leader_election.cc:302] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:13.261444 9131 raft_consensus.cc:2781] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:13.261533 9131 raft_consensus.cc:482] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:13.261624 9131 raft_consensus.cc:3037] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:13.262964 4087 catalog_manager.cc:5484] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: term changed from 1 to 2, leader changed from dca57823003e4729b769ca12367d7530 (127.29.219.65) to ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69). New cstate: current_term: 2 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:13.264437 9131 raft_consensus.cc:504] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:13.264750 9122 raft_consensus.cc:2932] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:13.266379 8790 consensus_queue.cc:1035] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:13.267297 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21f156318bd945dca004ae1afcc40a2c" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:13.267517 7155 raft_consensus.cc:3037] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:13.268431 4499 raft_consensus.cc:1203] T 53be6ced1df34676b1a34bc5cfe1ac87 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:13.270299 7155 raft_consensus.cc:2454] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:13.271997 7220 leader_election.cc:302] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:13.272341 9020 raft_consensus.cc:2781] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:13.272476 9020 raft_consensus.cc:686] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Becoming Leader. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Running, Role: LEADER
I20240501 20:10:13.272776 9020 consensus_queue.cc:237] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:13.272737 8877 consensus_queue.cc:1035] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 20:10:13.273375 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 56cb0964dabd41079d4428787efc9a9d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:13.281184 4088 catalog_manager.cc:5484] T 56cb0964dabd41079d4428787efc9a9d P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:13.281560 9159 consensus_queue.cc:1035] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:13.275575 4496 raft_consensus.cc:2932] T 56cb0964dabd41079d4428787efc9a9d P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:13.286180 8940 raft_consensus.cc:2932] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:13.290246 9065 raft_consensus.cc:482] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:13.290437 9065 raft_consensus.cc:504] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:13.291635 9065 leader_election.cc:288] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:13.292742 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21f156318bd945dca004ae1afcc40a2c" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:13.293054 7153 raft_consensus.cc:2379] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dca57823003e4729b769ca12367d7530 in current term 1: Already voted for candidate bceeae1c79964acabdae8ff771bf0c97 in this term.
I20240501 20:10:13.293195 4627 raft_consensus.cc:2932] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:13.293265 4233 raft_consensus.cc:2932] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:13.297878 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21f156318bd945dca004ae1afcc40a2c" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:13.298530 4168 leader_election.cc:302] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dca57823003e4729b769ca12367d7530; no voters: bceeae1c79964acabdae8ff771bf0c97, db0742ee3860468590eefef6ae6074fd
I20240501 20:10:13.277060 7155 raft_consensus.cc:2932] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:13.301998 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21f156318bd945dca004ae1afcc40a2c" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:13.302235 4234 raft_consensus.cc:2454] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:13.308939 4347 tablet_service.cc:1514] Processing DeleteTablet for tablet 56cb0964dabd41079d4428787efc9a9d with delete_type TABLET_DATA_TOMBSTONED (TS 10b47782558c41ee91f5e164a372049a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:13.309283 9180 tablet_replica.cc:330] T 56cb0964dabd41079d4428787efc9a9d P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:13.309506 9180 raft_consensus.cc:2229] T 56cb0964dabd41079d4428787efc9a9d P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:13.309703 9180 raft_consensus.cc:2258] T 56cb0964dabd41079d4428787efc9a9d P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:13.310900 9180 ts_tablet_manager.cc:1875] T 56cb0964dabd41079d4428787efc9a9d P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:13.311872 4091 catalog_manager.cc:5484] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:13.317166 9131 leader_election.cc:288] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Requested vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:13.318672 4088 catalog_manager.cc:5484] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71). New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:13.319882 8785 raft_consensus.cc:3037] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:13.322860 9180 ts_tablet_manager.cc:1888] T 56cb0964dabd41079d4428787efc9a9d P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:13.323019 9180 log.cc:1196] T 56cb0964dabd41079d4428787efc9a9d P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/56cb0964dabd41079d4428787efc9a9d
I20240501 20:10:13.322722 7419 raft_consensus.cc:2932] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:13.326831 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet 56cb0964dabd41079d4428787efc9a9d (table rebalance_test_table_15 [id=2478c1c300d14f2a9826930dc16a5dec]) successfully deleted
I20240501 20:10:13.336725 8785 raft_consensus.cc:2726] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:13.337582 4093 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:13.341753 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21f156318bd945dca004ae1afcc40a2c" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:10:13.345523 4233 raft_consensus.cc:2454] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:13.352257 7266 tablet_service.cc:1467] Processing CreateTablet for tablet b2619f3bed304d3ab864f56d5b655574 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=306aca4fdfa94f4f81f64f300d602707]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:13.352762 7266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2619f3bed304d3ab864f56d5b655574. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:13.353719 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 797b2374d4214c8eb39982c339a7fbad (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=306aca4fdfa94f4f81f64f300d602707]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:13.354151 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 797b2374d4214c8eb39982c339a7fbad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:13.354084 4347 tablet_service.cc:1467] Processing CreateTablet for tablet 26ed0760e9074bd896e55e99756602bb (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=306aca4fdfa94f4f81f64f300d602707]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:13.354489 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ed0760e9074bd896e55e99756602bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:13.356598 7135 tablet_service.cc:1467] Processing CreateTablet for tablet b2619f3bed304d3ab864f56d5b655574 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=306aca4fdfa94f4f81f64f300d602707]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:13.357041 7135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2619f3bed304d3ab864f56d5b655574. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:13.357012 4609 tablet_service.cc:1467] Processing CreateTablet for tablet b2619f3bed304d3ab864f56d5b655574 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=306aca4fdfa94f4f81f64f300d602707]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:13.357403 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2619f3bed304d3ab864f56d5b655574. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:13.359046 7134 tablet_service.cc:1467] Processing CreateTablet for tablet 26ed0760e9074bd896e55e99756602bb (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=306aca4fdfa94f4f81f64f300d602707]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:13.359405 7134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ed0760e9074bd896e55e99756602bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:13.363214 7133 tablet_service.cc:1467] Processing CreateTablet for tablet 797b2374d4214c8eb39982c339a7fbad (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=306aca4fdfa94f4f81f64f300d602707]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:13.363808 7133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 797b2374d4214c8eb39982c339a7fbad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:13.366089 9206 tablet_bootstrap.cc:492] T 26ed0760e9074bd896e55e99756602bb P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:13.368494 9206 tablet_bootstrap.cc:654] T 26ed0760e9074bd896e55e99756602bb P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:13.373533 9139 tablet_bootstrap.cc:492] T b2619f3bed304d3ab864f56d5b655574 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:13.375723 9139 tablet_bootstrap.cc:654] T b2619f3bed304d3ab864f56d5b655574 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:13.379329 7267 tablet_service.cc:1467] Processing CreateTablet for tablet 797b2374d4214c8eb39982c339a7fbad (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=306aca4fdfa94f4f81f64f300d602707]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:13.380023 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 797b2374d4214c8eb39982c339a7fbad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:13.383646 7265 tablet_service.cc:1467] Processing CreateTablet for tablet 26ed0760e9074bd896e55e99756602bb (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=306aca4fdfa94f4f81f64f300d602707]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:13.384182 7265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ed0760e9074bd896e55e99756602bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:13.389534 9139 tablet_bootstrap.cc:492] T b2619f3bed304d3ab864f56d5b655574 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:13.390753 9139 ts_tablet_manager.cc:1367] T b2619f3bed304d3ab864f56d5b655574 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20240501 20:10:13.391734 9139 raft_consensus.cc:348] T b2619f3bed304d3ab864f56d5b655574 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:13.392014 9139 raft_consensus.cc:374] T b2619f3bed304d3ab864f56d5b655574 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:13.392108 9139 raft_consensus.cc:729] T b2619f3bed304d3ab864f56d5b655574 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:13.391764 9229 tablet_bootstrap.cc:492] T 797b2374d4214c8eb39982c339a7fbad P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:13.393000 9139 consensus_queue.cc:260] T b2619f3bed304d3ab864f56d5b655574 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:13.397054 9162 raft_consensus.cc:482] T b2619f3bed304d3ab864f56d5b655574 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:13.397254 9162 raft_consensus.cc:504] T b2619f3bed304d3ab864f56d5b655574 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:13.398164 9162 leader_election.cc:288] T b2619f3bed304d3ab864f56d5b655574 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:13.398224 9229 tablet_bootstrap.cc:654] T 797b2374d4214c8eb39982c339a7fbad P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
W20240501 20:10:13.399698 4563 leader_election.cc:341] T b2619f3bed304d3ab864f56d5b655574 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Not found: Tablet not found: b2619f3bed304d3ab864f56d5b655574
I20240501 20:10:13.400907 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2619f3bed304d3ab864f56d5b655574" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:13.402025 9139 ts_tablet_manager.cc:1398] T b2619f3bed304d3ab864f56d5b655574 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
W20240501 20:10:13.402191 4562 leader_election.cc:341] T b2619f3bed304d3ab864f56d5b655574 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:13.402339 4562 leader_election.cc:302] T b2619f3bed304d3ab864f56d5b655574 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e; no voters: bceeae1c79964acabdae8ff771bf0c97, db0742ee3860468590eefef6ae6074fd
I20240501 20:10:13.403169 9162 raft_consensus.cc:2726] T b2619f3bed304d3ab864f56d5b655574 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:13.405105 9232 tablet_bootstrap.cc:492] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:13.408042 9232 tablet_bootstrap.cc:654] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:13.410665 9229 tablet_bootstrap.cc:492] T 797b2374d4214c8eb39982c339a7fbad P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:13.410907 9229 ts_tablet_manager.cc:1367] T 797b2374d4214c8eb39982c339a7fbad P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.006s
I20240501 20:10:13.412117 9229 raft_consensus.cc:348] T 797b2374d4214c8eb39982c339a7fbad P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:13.413734 9229 raft_consensus.cc:374] T 797b2374d4214c8eb39982c339a7fbad P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:13.414011 9229 raft_consensus.cc:729] T 797b2374d4214c8eb39982c339a7fbad P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:13.414113 9232 tablet_bootstrap.cc:492] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:10:13.414342 9232 ts_tablet_manager.cc:1367] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240501 20:10:13.414644 9229 consensus_queue.cc:260] T 797b2374d4214c8eb39982c339a7fbad P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:13.415438 9232 raft_consensus.cc:348] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:13.415681 9232 raft_consensus.cc:374] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:13.415946 9232 raft_consensus.cc:729] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:10:13.416828 9229 ts_tablet_manager.cc:1398] T 797b2374d4214c8eb39982c339a7fbad P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 20:10:13.417202 9232 consensus_queue.cc:260] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:13.419711 9232 ts_tablet_manager.cc:1398] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 20:10:13.432226 9128 tablet_bootstrap.cc:492] T 26ed0760e9074bd896e55e99756602bb P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:13.434569 9128 tablet_bootstrap.cc:654] T 26ed0760e9074bd896e55e99756602bb P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:13.435493 9206 tablet_bootstrap.cc:492] T 26ed0760e9074bd896e55e99756602bb P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:13.435698 9206 ts_tablet_manager.cc:1367] T 26ed0760e9074bd896e55e99756602bb P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.007s
I20240501 20:10:13.436702 9206 raft_consensus.cc:348] T 26ed0760e9074bd896e55e99756602bb P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:13.437120 9206 raft_consensus.cc:374] T 26ed0760e9074bd896e55e99756602bb P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:13.437235 9206 raft_consensus.cc:729] T 26ed0760e9074bd896e55e99756602bb P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:13.437532 9206 consensus_queue.cc:260] T 26ed0760e9074bd896e55e99756602bb P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:13.438129 9206 ts_tablet_manager.cc:1398] T 26ed0760e9074bd896e55e99756602bb P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:10:13.438511 9206 tablet_bootstrap.cc:492] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:13.440168 9206 tablet_bootstrap.cc:654] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:13.441730 9128 tablet_bootstrap.cc:492] T 26ed0760e9074bd896e55e99756602bb P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:13.442023 9128 ts_tablet_manager.cc:1367] T 26ed0760e9074bd896e55e99756602bb P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240501 20:10:13.444972 9128 raft_consensus.cc:348] T 26ed0760e9074bd896e55e99756602bb P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:13.445412 9128 raft_consensus.cc:374] T 26ed0760e9074bd896e55e99756602bb P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:13.445557 9128 raft_consensus.cc:729] T 26ed0760e9074bd896e55e99756602bb P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:13.445940 9128 consensus_queue.cc:260] T 26ed0760e9074bd896e55e99756602bb P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:13.451468 9206 tablet_bootstrap.cc:492] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:13.451701 9206 ts_tablet_manager.cc:1367] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20240501 20:10:13.451828 9128 ts_tablet_manager.cc:1398] T 26ed0760e9074bd896e55e99756602bb P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 20:10:13.452184 9128 tablet_bootstrap.cc:492] T b2619f3bed304d3ab864f56d5b655574 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:13.452948 9206 raft_consensus.cc:348] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:13.453193 9206 raft_consensus.cc:374] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:13.453299 9206 raft_consensus.cc:729] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:13.453791 9128 tablet_bootstrap.cc:654] T b2619f3bed304d3ab864f56d5b655574 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:13.454239 9206 consensus_queue.cc:260] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:13.455081 9206 ts_tablet_manager.cc:1398] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20240501 20:10:13.457491 9206 tablet_bootstrap.cc:492] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:13.460297 9206 tablet_bootstrap.cc:654] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:13.472586 9128 tablet_bootstrap.cc:492] T b2619f3bed304d3ab864f56d5b655574 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:13.472806 9128 ts_tablet_manager.cc:1367] T b2619f3bed304d3ab864f56d5b655574 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.013s
I20240501 20:10:13.473904 9128 raft_consensus.cc:348] T b2619f3bed304d3ab864f56d5b655574 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:13.474131 9128 raft_consensus.cc:374] T b2619f3bed304d3ab864f56d5b655574 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:13.474232 9128 raft_consensus.cc:729] T b2619f3bed304d3ab864f56d5b655574 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:13.474485 9128 consensus_queue.cc:260] T b2619f3bed304d3ab864f56d5b655574 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:13.475169 9128 ts_tablet_manager.cc:1398] T b2619f3bed304d3ab864f56d5b655574 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:10:13.475708 9128 tablet_bootstrap.cc:492] T 797b2374d4214c8eb39982c339a7fbad P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:13.477842 9128 tablet_bootstrap.cc:654] T 797b2374d4214c8eb39982c339a7fbad P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:13.480149 4091 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:13.481570 9206 tablet_bootstrap.cc:492] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:13.481793 9206 ts_tablet_manager.cc:1367] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.002s
I20240501 20:10:13.484499 9206 raft_consensus.cc:348] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:13.484858 9206 raft_consensus.cc:374] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:13.484935 9206 raft_consensus.cc:729] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:13.485158 9206 consensus_queue.cc:260] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:13.486821 9206 ts_tablet_manager.cc:1398] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240501 20:10:13.488212 9128 tablet_bootstrap.cc:492] T 797b2374d4214c8eb39982c339a7fbad P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:13.488462 9128 ts_tablet_manager.cc:1367] T 797b2374d4214c8eb39982c339a7fbad P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.007s
I20240501 20:10:13.488729 8604 raft_consensus.cc:482] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:13.488914 8604 raft_consensus.cc:504] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:13.489269 9128 raft_consensus.cc:348] T 797b2374d4214c8eb39982c339a7fbad P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:13.489470 9128 raft_consensus.cc:374] T 797b2374d4214c8eb39982c339a7fbad P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:13.489550 9128 raft_consensus.cc:729] T 797b2374d4214c8eb39982c339a7fbad P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:13.489656 8604 leader_election.cc:288] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:13.489840 9128 consensus_queue.cc:260] T 797b2374d4214c8eb39982c339a7fbad P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:13.490326 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ed0760e9074bd896e55e99756602bb" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:13.490478 9128 ts_tablet_manager.cc:1398] T 797b2374d4214c8eb39982c339a7fbad P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:10:13.490559 7155 raft_consensus.cc:2454] T 26ed0760e9074bd896e55e99756602bb P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:10:13.491107 4299 leader_election.cc:302] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:13.491191 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ed0760e9074bd896e55e99756602bb" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:13.491447 7285 raft_consensus.cc:2454] T 26ed0760e9074bd896e55e99756602bb P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:10:13.491561 8604 raft_consensus.cc:2781] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:13.491652 8604 raft_consensus.cc:482] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:13.491753 8604 raft_consensus.cc:3037] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:13.494503 8604 raft_consensus.cc:504] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:13.495666 8604 leader_election.cc:288] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Requested vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:13.496098 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ed0760e9074bd896e55e99756602bb" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:13.496326 7155 raft_consensus.cc:3037] T 26ed0760e9074bd896e55e99756602bb P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:13.498297 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26ed0760e9074bd896e55e99756602bb" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:13.498554 7285 raft_consensus.cc:3037] T 26ed0760e9074bd896e55e99756602bb P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:13.499253 7155 raft_consensus.cc:2454] T 26ed0760e9074bd896e55e99756602bb P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:10:13.499897 4233 raft_consensus.cc:1261] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:13.500087 4299 leader_election.cc:302] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:13.500365 8604 raft_consensus.cc:2781] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:13.500658 9145 consensus_queue.cc:1035] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:13.500524 8604 raft_consensus.cc:686] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Becoming Leader. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Running, Role: LEADER
I20240501 20:10:13.501052 7285 raft_consensus.cc:2454] T 26ed0760e9074bd896e55e99756602bb P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:10:13.501055 8604 consensus_queue.cc:237] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:13.501675 7153 raft_consensus.cc:1261] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:13.503582 9145 consensus_queue.cc:1035] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:13.514176 4091 catalog_manager.cc:5576] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941) does not have the latest schema for tablet 21f156318bd945dca004ae1afcc40a2c (table rebalancer_dynamic_table [id=ce784d0bfe4b43798e9861c3fa5866c2]). Expected version 1 got 0
I20240501 20:10:13.514981 4088 catalog_manager.cc:5484] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a reported cstate change: term changed from 0 to 1, leader changed from <none> to 10b47782558c41ee91f5e164a372049a (127.29.219.66). New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:13.524127 9257 tablet.cc:1722] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:13.524886 9257 tablet.cc:1620] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530: MemRowSet was empty: no flush needed.
I20240501 20:10:13.523377 9256 tablet.cc:1722] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:13.530452 9256 tablet.cc:1620] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:13.535264 4088 catalog_manager.cc:5576] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941) does not have the latest schema for tablet 21f156318bd945dca004ae1afcc40a2c (table rebalancer_dynamic_table [id=ce784d0bfe4b43798e9861c3fa5866c2]). Expected version 1 got 0
I20240501 20:10:13.542109 4091 catalog_manager.cc:5576] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447) does not have the latest schema for tablet 21f156318bd945dca004ae1afcc40a2c (table rebalancer_dynamic_table [id=ce784d0bfe4b43798e9861c3fa5866c2]). Expected version 1 got 0
I20240501 20:10:13.543603 4093 catalog_manager.cc:5576] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321) does not have the latest schema for tablet 21f156318bd945dca004ae1afcc40a2c (table rebalancer_dynamic_table [id=ce784d0bfe4b43798e9861c3fa5866c2]). Expected version 1 got 0
I20240501 20:10:13.551239 4091 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=ce784d0bfe4b43798e9861c3fa5866c2] alter complete (version 1)
I20240501 20:10:13.553892 4093 catalog_manager.cc:5576] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321) does not have the latest schema for tablet 21f156318bd945dca004ae1afcc40a2c (table rebalancer_dynamic_table [id=ce784d0bfe4b43798e9861c3fa5866c2]). Expected version 1 got 0
I20240501 20:10:13.557317 9264 tablet.cc:1722] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:13.557925 9264 tablet.cc:1620] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:10:13.577500 8139 raft_consensus.cc:482] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:13.577708 8139 raft_consensus.cc:504] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:13.578478 8139 leader_election.cc:288] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:13.579016 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "797b2374d4214c8eb39982c339a7fbad" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:13.579047 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "797b2374d4214c8eb39982c339a7fbad" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:13.579263 7420 raft_consensus.cc:2454] T 797b2374d4214c8eb39982c339a7fbad P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:13.579280 7285 raft_consensus.cc:2454] T 797b2374d4214c8eb39982c339a7fbad P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:13.579875 7091 leader_election.cc:302] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:13.580319 8139 raft_consensus.cc:2781] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:13.580415 8139 raft_consensus.cc:482] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:13.580515 8139 raft_consensus.cc:3037] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:13.584650 8139 raft_consensus.cc:504] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:13.586361 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "797b2374d4214c8eb39982c339a7fbad" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:13.586648 7420 raft_consensus.cc:3037] T 797b2374d4214c8eb39982c339a7fbad P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:13.589231 7420 raft_consensus.cc:2454] T 797b2374d4214c8eb39982c339a7fbad P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:13.590166 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "797b2374d4214c8eb39982c339a7fbad" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:13.590375 7285 raft_consensus.cc:3037] T 797b2374d4214c8eb39982c339a7fbad P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:13.590781 7091 leader_election.cc:302] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:13.591081 9201 raft_consensus.cc:2781] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:13.591212 9201 raft_consensus.cc:686] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Becoming Leader. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Running, Role: LEADER
I20240501 20:10:13.591526 9201 consensus_queue.cc:237] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:13.593747 7285 raft_consensus.cc:2454] T 797b2374d4214c8eb39982c339a7fbad P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:13.595515 8139 leader_election.cc:288] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Requested vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:13.601928 8139 raft_consensus.cc:482] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:13.602283 8139 raft_consensus.cc:504] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:13.603721 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2619f3bed304d3ab864f56d5b655574" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:13.604137 7285 raft_consensus.cc:2454] T b2619f3bed304d3ab864f56d5b655574 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:13.604694 7089 leader_election.cc:302] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:13.605098 9033 raft_consensus.cc:2781] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:13.605227 9033 raft_consensus.cc:482] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:13.605334 9033 raft_consensus.cc:3037] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:13.609516 9033 raft_consensus.cc:504] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:13.612017 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2619f3bed304d3ab864f56d5b655574" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:13.612278 7287 raft_consensus.cc:3037] T b2619f3bed304d3ab864f56d5b655574 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:13.616921 7287 raft_consensus.cc:2454] T b2619f3bed304d3ab864f56d5b655574 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:13.617792 7089 leader_election.cc:302] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:13.618983 9033 leader_election.cc:288] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Requested vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:13.619374 4630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2619f3bed304d3ab864f56d5b655574" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:10:13.619664 4630 raft_consensus.cc:3037] T b2619f3bed304d3ab864f56d5b655574 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:13.620082 9033 raft_consensus.cc:2781] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:13.620223 9033 raft_consensus.cc:686] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Becoming Leader. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Running, Role: LEADER
I20240501 20:10:13.620514 9033 consensus_queue.cc:237] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:13.621891 8139 leader_election.cc:288] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:13.628158 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2619f3bed304d3ab864f56d5b655574" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:13.629876 4630 raft_consensus.cc:2454] T b2619f3bed304d3ab864f56d5b655574 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:13.633733 4093 catalog_manager.cc:5484] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd reported cstate change: term changed from 0 to 1, leader changed from <none> to db0742ee3860468590eefef6ae6074fd (127.29.219.70). New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:13.642115 4093 catalog_manager.cc:5484] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd reported cstate change: term changed from 0 to 1, leader changed from <none> to db0742ee3860468590eefef6ae6074fd (127.29.219.70). New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:13.647935 4630 raft_consensus.cc:1261] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:10:13.648808 8877 consensus_queue.cc:1035] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:13.655205 4093 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:13.664764 9256 tablet.cc:1722] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:13.665351 9256 tablet.cc:1620] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:13.665977 9264 tablet.cc:1722] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:13.666445 9264 tablet.cc:1620] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:10:13.667817 9257 tablet.cc:1722] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:13.668291 9257 tablet.cc:1620] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530: MemRowSet was empty: no flush needed.
I20240501 20:10:13.674038 4496 raft_consensus.cc:1261] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:10:13.675189 8877 consensus_queue.cc:1035] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:13.685787 4093 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=ce784d0bfe4b43798e9861c3fa5866c2] alter complete (version 2)
I20240501 20:10:13.707373 9065 raft_consensus.cc:1050] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530: attempting to promote NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b to VOTER
I20240501 20:10:13.708566 9065 consensus_queue.cc:237] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:13.711498 4499 raft_consensus.cc:1261] T 53be6ced1df34676b1a34bc5cfe1ac87 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:13.712208 8790 consensus_queue.cc:1035] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:13.713008 7419 raft_consensus.cc:1261] T 53be6ced1df34676b1a34bc5cfe1ac87 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:13.713773 7285 raft_consensus.cc:1261] T 53be6ced1df34676b1a34bc5cfe1ac87 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:13.714777 8785 consensus_queue.cc:1035] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:13.715339 8785 consensus_queue.cc:1035] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:13.738518 4233 raft_consensus.cc:1261] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [term 2 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:10:13.738664 7022 consensus_queue.cc:237] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:13.744189 4630 raft_consensus.cc:1261] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:10:13.745901 9158 consensus_queue.cc:1035] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:13.753477 8877 raft_consensus.cc:2932] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:13.754365 4630 raft_consensus.cc:2932] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:13.760099 8785 raft_consensus.cc:2932] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:13.761514 4499 raft_consensus.cc:2932] T 53be6ced1df34676b1a34bc5cfe1ac87 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:13.761785 4093 catalog_manager.cc:5484] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 5 to 7, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New cstate: current_term: 2 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:13.766180 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f18af954cefa48d4882313fe18abbd27 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:13.770946 4498 raft_consensus.cc:1261] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:10:13.773532 7420 raft_consensus.cc:2932] T 53be6ced1df34676b1a34bc5cfe1ac87 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:13.774242 4087 catalog_manager.cc:5484] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:13.776501 9158 consensus_queue.cc:1035] T f18af954cefa48d4882313fe18abbd27 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:13.778474 4498 raft_consensus.cc:2932] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:13.780575 4233 consensus_queue.cc:237] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:13.783278 7420 raft_consensus.cc:1261] T 53be6ced1df34676b1a34bc5cfe1ac87 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:13.784296 8790 consensus_queue.cc:1035] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:13.788136 8790 raft_consensus.cc:2932] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:13.794960 7419 raft_consensus.cc:2932] T 53be6ced1df34676b1a34bc5cfe1ac87 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:13.796975 4092 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:13.799320 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 53be6ced1df34676b1a34bc5cfe1ac87 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:13.802404 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet f18af954cefa48d4882313fe18abbd27 with delete_type TABLET_DATA_TOMBSTONED (TS dca57823003e4729b769ca12367d7530 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:13.807296 4088 catalog_manager.cc:5484] T 53be6ced1df34676b1a34bc5cfe1ac87 P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:13.809478 9256 tablet.cc:1722] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:13.814513 9257 tablet.cc:1722] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:13.816375 9264 tablet.cc:1722] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:13.819895 7265 tablet_service.cc:1514] Processing DeleteTablet for tablet 53be6ced1df34676b1a34bc5cfe1ac87 with delete_type TABLET_DATA_TOMBSTONED (TS bceeae1c79964acabdae8ff771bf0c97 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54166
I20240501 20:10:13.828868 4093 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table_new_name [id=ce784d0bfe4b43798e9861c3fa5866c2] alter complete (version 3)
I20240501 20:10:13.840893 4497 raft_consensus.cc:1261] T 53be6ced1df34676b1a34bc5cfe1ac87 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:13.841059 9294 tablet_replica.cc:330] T 53be6ced1df34676b1a34bc5cfe1ac87 P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:13.841281 9294 raft_consensus.cc:2229] T 53be6ced1df34676b1a34bc5cfe1ac87 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:13.841470 9294 pending_rounds.cc:62] T 53be6ced1df34676b1a34bc5cfe1ac87 P bceeae1c79964acabdae8ff771bf0c97: Trying to abort 1 pending ops.
I20240501 20:10:13.841810 8790 consensus_queue.cc:1035] T 53be6ced1df34676b1a34bc5cfe1ac87 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:13.841578 9294 pending_rounds.cc:68] T 53be6ced1df34676b1a34bc5cfe1ac87 P bceeae1c79964acabdae8ff771bf0c97: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7022977899347263488 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "53be6ced1df34676b1a34bc5cfe1ac87" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } } }
I20240501 20:10:13.842080 9294 raft_consensus.cc:2894] T 53be6ced1df34676b1a34bc5cfe1ac87 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240501 20:10:13.842218 9294 raft_consensus.cc:2866] T 53be6ced1df34676b1a34bc5cfe1ac87 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:10:13.842336 9294 raft_consensus.cc:2258] T 53be6ced1df34676b1a34bc5cfe1ac87 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:13.843340 4499 raft_consensus.cc:2932] T 53be6ced1df34676b1a34bc5cfe1ac87 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:13.845044 9293 tablet_replica.cc:330] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:13.845702 9293 raft_consensus.cc:2229] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:13.846205 9293 raft_consensus.cc:2258] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:13.847077 9294 ts_tablet_manager.cc:1875] T 53be6ced1df34676b1a34bc5cfe1ac87 P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:13.848618 9293 ts_tablet_manager.cc:1875] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:13.853691 9294 ts_tablet_manager.cc:1888] T 53be6ced1df34676b1a34bc5cfe1ac87 P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:13.853978 9294 log.cc:1196] T 53be6ced1df34676b1a34bc5cfe1ac87 P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/53be6ced1df34676b1a34bc5cfe1ac87
I20240501 20:10:13.854907 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet 53be6ced1df34676b1a34bc5cfe1ac87 (table rebalancer_extra_table_2 [id=983618faca8342e4aea12a20efab90c0]) successfully deleted
I20240501 20:10:13.866019 9293 ts_tablet_manager.cc:1888] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:13.866261 9293 log.cc:1196] T f18af954cefa48d4882313fe18abbd27 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/f18af954cefa48d4882313fe18abbd27
I20240501 20:10:13.867256 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet f18af954cefa48d4882313fe18abbd27 (table rebalance_test_table_3 [id=5fb9ba92237649f89c99e4487cd45f29]) successfully deleted
I20240501 20:10:13.880940 4365 consensus_queue.cc:237] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:13.883937 4234 raft_consensus.cc:1261] T 91a2bf5ddc03431b963ab5da831fd340 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:13.883980 4499 raft_consensus.cc:1261] T 91a2bf5ddc03431b963ab5da831fd340 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:13.884279 4302 consensus_peers.cc:485] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a -> Peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): Couldn't send request to peer b9957d4086bc4a76b18ed9265a5739d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91a2bf5ddc03431b963ab5da831fd340. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:13.884644 8604 consensus_queue.cc:1035] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:13.885313 9143 consensus_queue.cc:1035] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:13.887755 8604 raft_consensus.cc:2932] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:13.892885 4234 raft_consensus.cc:2932] T 91a2bf5ddc03431b963ab5da831fd340 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:13.895550 4499 raft_consensus.cc:2932] T 91a2bf5ddc03431b963ab5da831fd340 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:13.896024 4093 catalog_manager.cc:5484] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:13.928254 4499 consensus_queue.cc:237] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
W20240501 20:10:13.931217 4434 consensus_peers.cc:485] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a20287c2cab44a62b28037a463d4b49b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:13.931201 4365 raft_consensus.cc:1261] T a20287c2cab44a62b28037a463d4b49b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:13.932114 9169 consensus_queue.cc:1035] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:13.932804 4630 raft_consensus.cc:1261] T a20287c2cab44a62b28037a463d4b49b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:13.933646 8940 consensus_queue.cc:1035] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:13.934405 4093 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:13.934965 4093 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:13.939347 4093 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 21f156318bd945dca004ae1afcc40a2c
I20240501 20:10:13.939054 8940 raft_consensus.cc:2932] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:13.940166 7265 tablet_service.cc:1514] Processing DeleteTablet for tablet 21f156318bd945dca004ae1afcc40a2c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:13 UTC) from {username='slave'} at 127.0.0.1:54166
I20240501 20:10:13.940515 9294 tablet_replica.cc:330] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:13.940505 7133 tablet_service.cc:1514] Processing DeleteTablet for tablet 21f156318bd945dca004ae1afcc40a2c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:13 UTC) from {username='slave'} at 127.0.0.1:52036
I20240501 20:10:13.940745 9294 raft_consensus.cc:2229] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:13.941135 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet 21f156318bd945dca004ae1afcc40a2c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:13 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:13.941182 9294 raft_consensus.cc:2258] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:13.941392 9317 tablet_replica.cc:330] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd: stopping tablet replica
I20240501 20:10:13.941537 9293 tablet_replica.cc:330] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:13.941627 9317 raft_consensus.cc:2229] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:13.941753 9293 raft_consensus.cc:2229] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:13.941851 9317 raft_consensus.cc:2258] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:13.941951 9293 raft_consensus.cc:2258] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:13.942868 9317 ts_tablet_manager.cc:1875] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:13.943071 9293 ts_tablet_manager.cc:1875] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:13.943398 9294 ts_tablet_manager.cc:1875] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:13.943908 4630 raft_consensus.cc:2932] T a20287c2cab44a62b28037a463d4b49b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:13.946645 4365 raft_consensus.cc:2932] T a20287c2cab44a62b28037a463d4b49b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:13.951520 9293 ts_tablet_manager.cc:1888] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:13.953910 9293 log.cc:1196] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/21f156318bd945dca004ae1afcc40a2c
I20240501 20:10:13.954524 9293 ts_tablet_manager.cc:1909] T 21f156318bd945dca004ae1afcc40a2c P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:13.956117 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet 21f156318bd945dca004ae1afcc40a2c (table rebalancer_dynamic_table_new_name [id=ce784d0bfe4b43798e9861c3fa5866c2]) successfully deleted
I20240501 20:10:13.955366 4093 catalog_manager.cc:5484] T a20287c2cab44a62b28037a463d4b49b P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index -1 to 4, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:13.962446 9317 ts_tablet_manager.cc:1888] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:13.962607 9317 log.cc:1196] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/wals/21f156318bd945dca004ae1afcc40a2c
I20240501 20:10:13.963094 9317 ts_tablet_manager.cc:1909] T 21f156318bd945dca004ae1afcc40a2c P db0742ee3860468590eefef6ae6074fd: Deleting consensus metadata
I20240501 20:10:13.964022 9294 ts_tablet_manager.cc:1888] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:13.964068 4076 catalog_manager.cc:4830] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): tablet 21f156318bd945dca004ae1afcc40a2c (table rebalancer_dynamic_table_new_name [id=ce784d0bfe4b43798e9861c3fa5866c2]) successfully deleted
I20240501 20:10:13.964201 9294 log.cc:1196] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/21f156318bd945dca004ae1afcc40a2c
I20240501 20:10:13.964792 9294 ts_tablet_manager.cc:1909] T 21f156318bd945dca004ae1afcc40a2c P bceeae1c79964acabdae8ff771bf0c97: Deleting consensus metadata
I20240501 20:10:13.966142 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet 21f156318bd945dca004ae1afcc40a2c (table rebalancer_dynamic_table_new_name [id=ce784d0bfe4b43798e9861c3fa5866c2]) successfully deleted
I20240501 20:10:13.983620 4630 consensus_queue.cc:237] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:13.984611 9033 consensus_queue.cc:1035] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:13.986436 4499 raft_consensus.cc:1261] T 23eaac7a50354947b6efa26ac5f01ae7 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:13.986987 4563 consensus_peers.cc:485] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Couldn't send request to peer bceeae1c79964acabdae8ff771bf0c97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23eaac7a50354947b6efa26ac5f01ae7. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:13.987918 9162 consensus_queue.cc:1035] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:13.991989 8139 consensus_queue.cc:1035] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:13.995999 4367 raft_consensus.cc:1261] T 23eaac7a50354947b6efa26ac5f01ae7 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:13.997591 9326 consensus_queue.cc:1035] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:14.002310 9326 raft_consensus.cc:2932] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:14.005563 4499 raft_consensus.cc:2932] T 23eaac7a50354947b6efa26ac5f01ae7 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:14.011787 4093 catalog_manager.cc:5484] T 23eaac7a50354947b6efa26ac5f01ae7 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:14.012743 4367 raft_consensus.cc:2932] T 23eaac7a50354947b6efa26ac5f01ae7 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:14.023768 9306 consensus_queue.cc:1035] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:14.030236 9143 consensus_queue.cc:1035] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:14.042816 4092 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:14.047394 4234 consensus_queue.cc:237] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:14.050134 4630 raft_consensus.cc:1261] T a8eefa22b6e74b92b9fef036dea46e3b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:14.051077 4167 consensus_peers.cc:485] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 -> Peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Couldn't send request to peer db0742ee3860468590eefef6ae6074fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a8eefa22b6e74b92b9fef036dea46e3b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:14.051126 8790 consensus_queue.cc:1035] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:14.052167 7133 tablet_service.cc:1467] Processing CreateTablet for tablet 91193b2f5c1c4fc8ad4a443f73c75ed8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ea651d4353a24bc3a854da0192253c56]), partition=
I20240501 20:10:14.052175 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 91193b2f5c1c4fc8ad4a443f73c75ed8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ea651d4353a24bc3a854da0192253c56]), partition=
I20240501 20:10:14.052198 4497 raft_consensus.cc:1261] T a8eefa22b6e74b92b9fef036dea46e3b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:14.052206 7002 tablet_service.cc:1467] Processing CreateTablet for tablet 91193b2f5c1c4fc8ad4a443f73c75ed8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ea651d4353a24bc3a854da0192253c56]), partition=
I20240501 20:10:14.052531 7133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91193b2f5c1c4fc8ad4a443f73c75ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:14.052549 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91193b2f5c1c4fc8ad4a443f73c75ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:14.052615 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91193b2f5c1c4fc8ad4a443f73c75ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:14.052875 8785 consensus_queue.cc:1035] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:14.066077 8790 raft_consensus.cc:2932] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:14.067677 9345 tablet_bootstrap.cc:492] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:14.068439 9344 tablet_bootstrap.cc:492] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:14.069409 9345 tablet_bootstrap.cc:654] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:14.073089 9346 tablet_bootstrap.cc:492] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:14.073863 9344 tablet_bootstrap.cc:654] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:14.074292 4499 raft_consensus.cc:2932] T a8eefa22b6e74b92b9fef036dea46e3b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:14.077040 4627 raft_consensus.cc:2932] T a8eefa22b6e74b92b9fef036dea46e3b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:14.079133 9346 tablet_bootstrap.cc:654] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:14.083619 9346 tablet_bootstrap.cc:492] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:14.083856 9346 ts_tablet_manager.cc:1367] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20240501 20:10:14.084861 9346 raft_consensus.cc:348] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:14.085270 9346 raft_consensus.cc:374] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:14.085439 9346 raft_consensus.cc:729] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:14.085753 9346 consensus_queue.cc:260] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:14.088991 4092 catalog_manager.cc:5484] T a8eefa22b6e74b92b9fef036dea46e3b P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:14.094899 9344 tablet_bootstrap.cc:492] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:14.095139 9344 ts_tablet_manager.cc:1367] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.000s
I20240501 20:10:14.096009 9344 raft_consensus.cc:348] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:14.097954 9344 raft_consensus.cc:374] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:14.098131 9344 raft_consensus.cc:729] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:14.098459 9344 consensus_queue.cc:260] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:14.100132 9344 ts_tablet_manager.cc:1398] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 20:10:14.100912 9345 tablet_bootstrap.cc:492] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:14.101114 9345 ts_tablet_manager.cc:1367] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.005s
I20240501 20:10:14.102082 9345 raft_consensus.cc:348] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:14.102349 9345 raft_consensus.cc:374] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:14.102481 9345 raft_consensus.cc:729] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:14.105631 9345 consensus_queue.cc:260] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:14.108109 9345 ts_tablet_manager.cc:1398] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240501 20:10:14.111517 9346 ts_tablet_manager.cc:1398] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
I20240501 20:10:14.125617 9201 consensus_queue.cc:1035] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:14.136406 7287 consensus_queue.cc:237] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:14.148763 7022 raft_consensus.cc:1261] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:14.149245 7222 consensus_peers.cc:485] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f82a24aabb2b4386b5916653a97e548c. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:14.149488 7419 raft_consensus.cc:1261] T f82a24aabb2b4386b5916653a97e548c P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:14.149616 9145 consensus_queue.cc:1035] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:14.155709 9033 consensus_queue.cc:1035] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:14.156193 9131 consensus_queue.cc:1035] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:14.167073 9145 raft_consensus.cc:2932] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:14.167949 7419 raft_consensus.cc:2932] T f82a24aabb2b4386b5916653a97e548c P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:14.174108 7022 raft_consensus.cc:2932] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:14.174520 4093 catalog_manager.cc:5484] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:14.204543 4091 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:14.221307 4478 tablet_service.cc:1467] Processing CreateTablet for tablet 185ee8ce5e7f4617b0526b5f3fd64a29 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d841e2e5903247b28bb65617a30e22cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:14.221592 7265 tablet_service.cc:1467] Processing CreateTablet for tablet 185ee8ce5e7f4617b0526b5f3fd64a29 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d841e2e5903247b28bb65617a30e22cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:14.221863 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 185ee8ce5e7f4617b0526b5f3fd64a29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:14.222060 7265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 185ee8ce5e7f4617b0526b5f3fd64a29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:14.224157 7002 tablet_service.cc:1467] Processing CreateTablet for tablet 129322eab07149d2a6fb7eef96100e7a (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d841e2e5903247b28bb65617a30e22cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:14.224601 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 129322eab07149d2a6fb7eef96100e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:14.225620 4347 tablet_service.cc:1467] Processing CreateTablet for tablet 185ee8ce5e7f4617b0526b5f3fd64a29 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d841e2e5903247b28bb65617a30e22cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:14.226054 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 185ee8ce5e7f4617b0526b5f3fd64a29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:14.230580 7267 tablet_service.cc:1467] Processing CreateTablet for tablet 129322eab07149d2a6fb7eef96100e7a (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d841e2e5903247b28bb65617a30e22cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:14.230841 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 129322eab07149d2a6fb7eef96100e7a (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d841e2e5903247b28bb65617a30e22cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:14.231076 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 129322eab07149d2a6fb7eef96100e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:14.231287 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 129322eab07149d2a6fb7eef96100e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:14.240070 7133 tablet_service.cc:1467] Processing CreateTablet for tablet 024c925714cb419aaa8870a053a6ca2f (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d841e2e5903247b28bb65617a30e22cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:14.240550 7133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 024c925714cb419aaa8870a053a6ca2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:14.241021 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 024c925714cb419aaa8870a053a6ca2f (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d841e2e5903247b28bb65617a30e22cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:14.241513 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 024c925714cb419aaa8870a053a6ca2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:14.241456 7266 tablet_service.cc:1467] Processing CreateTablet for tablet 024c925714cb419aaa8870a053a6ca2f (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=d841e2e5903247b28bb65617a30e22cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:14.241889 7266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 024c925714cb419aaa8870a053a6ca2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:14.249622 9345 tablet_bootstrap.cc:492] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:14.251408 9345 tablet_bootstrap.cc:654] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:14.251410 9346 tablet_bootstrap.cc:492] T 129322eab07149d2a6fb7eef96100e7a P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:14.253150 9346 tablet_bootstrap.cc:654] T 129322eab07149d2a6fb7eef96100e7a P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:14.254819 9344 tablet_bootstrap.cc:492] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:14.256955 9344 tablet_bootstrap.cc:654] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:14.299687 9344 tablet_bootstrap.cc:492] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:14.300045 9344 ts_tablet_manager.cc:1367] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.045s user 0.000s sys 0.011s
I20240501 20:10:14.301563 9369 tablet_bootstrap.cc:492] T 024c925714cb419aaa8870a053a6ca2f P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:14.302162 9344 raft_consensus.cc:348] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:14.302450 9344 raft_consensus.cc:374] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:14.302542 9344 raft_consensus.cc:729] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:14.302861 9344 consensus_queue.cc:260] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:14.304033 9367 tablet_bootstrap.cc:492] T 185ee8ce5e7f4617b0526b5f3fd64a29 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:14.304821 9344 ts_tablet_manager.cc:1398] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 20:10:14.306666 9345 tablet_bootstrap.cc:492] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:14.306785 9346 tablet_bootstrap.cc:492] T 129322eab07149d2a6fb7eef96100e7a P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:14.306944 9345 ts_tablet_manager.cc:1367] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.057s user 0.007s sys 0.014s
I20240501 20:10:14.307567 9366 tablet_bootstrap.cc:492] T 185ee8ce5e7f4617b0526b5f3fd64a29 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:14.307899 9346 ts_tablet_manager.cc:1367] T 129322eab07149d2a6fb7eef96100e7a P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.057s user 0.007s sys 0.004s
I20240501 20:10:14.308036 9345 raft_consensus.cc:348] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:14.308480 9345 raft_consensus.cc:374] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:14.308633 9345 raft_consensus.cc:729] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:14.308741 9369 tablet_bootstrap.cc:654] T 024c925714cb419aaa8870a053a6ca2f P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:14.308972 9345 consensus_queue.cc:260] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:14.309644 9345 ts_tablet_manager.cc:1398] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:10:14.310168 9366 tablet_bootstrap.cc:654] T 185ee8ce5e7f4617b0526b5f3fd64a29 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:14.311022 9367 tablet_bootstrap.cc:654] T 185ee8ce5e7f4617b0526b5f3fd64a29 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:14.311453 9346 raft_consensus.cc:348] T 129322eab07149d2a6fb7eef96100e7a P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:14.311702 9346 raft_consensus.cc:374] T 129322eab07149d2a6fb7eef96100e7a P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:14.311791 9346 raft_consensus.cc:729] T 129322eab07149d2a6fb7eef96100e7a P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:14.312064 9346 consensus_queue.cc:260] T 129322eab07149d2a6fb7eef96100e7a P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:14.312642 9346 ts_tablet_manager.cc:1398] T 129322eab07149d2a6fb7eef96100e7a P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:14.328094 9369 tablet_bootstrap.cc:492] T 024c925714cb419aaa8870a053a6ca2f P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:14.328131 9368 tablet_bootstrap.cc:492] T 129322eab07149d2a6fb7eef96100e7a P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:14.328315 9369 ts_tablet_manager.cc:1367] T 024c925714cb419aaa8870a053a6ca2f P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20240501 20:10:14.329161 9369 raft_consensus.cc:348] T 024c925714cb419aaa8870a053a6ca2f P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:14.329540 9369 raft_consensus.cc:374] T 024c925714cb419aaa8870a053a6ca2f P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:14.329651 9369 raft_consensus.cc:729] T 024c925714cb419aaa8870a053a6ca2f P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:14.329917 9369 consensus_queue.cc:260] T 024c925714cb419aaa8870a053a6ca2f P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:14.330904 9369 ts_tablet_manager.cc:1398] T 024c925714cb419aaa8870a053a6ca2f P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:14.331389 9367 tablet_bootstrap.cc:492] T 185ee8ce5e7f4617b0526b5f3fd64a29 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:10:14.331616 9367 ts_tablet_manager.cc:1367] T 185ee8ce5e7f4617b0526b5f3fd64a29 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20240501 20:10:14.332612 9367 raft_consensus.cc:348] T 185ee8ce5e7f4617b0526b5f3fd64a29 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:14.332978 9367 raft_consensus.cc:374] T 185ee8ce5e7f4617b0526b5f3fd64a29 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:14.333149 9367 raft_consensus.cc:729] T 185ee8ce5e7f4617b0526b5f3fd64a29 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:10:14.333227 9368 tablet_bootstrap.cc:654] T 129322eab07149d2a6fb7eef96100e7a P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:14.333482 9367 consensus_queue.cc:260] T 185ee8ce5e7f4617b0526b5f3fd64a29 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:14.334275 9367 ts_tablet_manager.cc:1398] T 185ee8ce5e7f4617b0526b5f3fd64a29 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:14.335620 9366 tablet_bootstrap.cc:492] T 185ee8ce5e7f4617b0526b5f3fd64a29 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:10:14.335821 9366 ts_tablet_manager.cc:1367] T 185ee8ce5e7f4617b0526b5f3fd64a29 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.003s
I20240501 20:10:14.336683 9366 raft_consensus.cc:348] T 185ee8ce5e7f4617b0526b5f3fd64a29 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:14.336954 9366 raft_consensus.cc:374] T 185ee8ce5e7f4617b0526b5f3fd64a29 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:14.337069 9366 raft_consensus.cc:729] T 185ee8ce5e7f4617b0526b5f3fd64a29 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:10:14.337356 9366 consensus_queue.cc:260] T 185ee8ce5e7f4617b0526b5f3fd64a29 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:14.339315 9366 ts_tablet_manager.cc:1398] T 185ee8ce5e7f4617b0526b5f3fd64a29 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:10:14.349774 9368 tablet_bootstrap.cc:492] T 129322eab07149d2a6fb7eef96100e7a P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:14.349994 9368 ts_tablet_manager.cc:1367] T 129322eab07149d2a6fb7eef96100e7a P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240501 20:10:14.350682 9368 raft_consensus.cc:348] T 129322eab07149d2a6fb7eef96100e7a P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:14.350862 9368 raft_consensus.cc:374] T 129322eab07149d2a6fb7eef96100e7a P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:14.350970 9368 raft_consensus.cc:729] T 129322eab07149d2a6fb7eef96100e7a P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:14.351212 9368 consensus_queue.cc:260] T 129322eab07149d2a6fb7eef96100e7a P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:14.351882 9368 ts_tablet_manager.cc:1398] T 129322eab07149d2a6fb7eef96100e7a P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:10:14.352242 9368 tablet_bootstrap.cc:492] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:14.353936 9368 tablet_bootstrap.cc:654] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:14.360139 9033 raft_consensus.cc:482] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:14.360352 9033 raft_consensus.cc:504] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:14.361080 9033 leader_election.cc:288] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:14.362057 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024c925714cb419aaa8870a053a6ca2f" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
W20240501 20:10:14.362663 7089 leader_election.cc:341] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:14.363910 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024c925714cb419aaa8870a053a6ca2f" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:14.364174 4234 raft_consensus.cc:2454] T 024c925714cb419aaa8870a053a6ca2f P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:14.364689 7090 leader_election.cc:302] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db0742ee3860468590eefef6ae6074fd, dca57823003e4729b769ca12367d7530; no voters: bceeae1c79964acabdae8ff771bf0c97
I20240501 20:10:14.365285 8139 raft_consensus.cc:2781] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:14.365413 8139 raft_consensus.cc:482] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:14.365518 8139 raft_consensus.cc:3037] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:14.368844 8139 raft_consensus.cc:504] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:14.370510 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024c925714cb419aaa8870a053a6ca2f" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:10:14.370784 4234 raft_consensus.cc:3037] T 024c925714cb419aaa8870a053a6ca2f P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:14.371521 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024c925714cb419aaa8870a053a6ca2f" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
W20240501 20:10:14.372094 7089 leader_election.cc:341] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:14.372274 8139 leader_election.cc:288] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Requested vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:14.373811 4234 raft_consensus.cc:2454] T 024c925714cb419aaa8870a053a6ca2f P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:14.374464 7090 leader_election.cc:302] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: db0742ee3860468590eefef6ae6074fd, dca57823003e4729b769ca12367d7530; no voters: bceeae1c79964acabdae8ff771bf0c97
I20240501 20:10:14.374894 8139 raft_consensus.cc:2781] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:14.375548 8139 raft_consensus.cc:686] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Becoming Leader. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Running, Role: LEADER
I20240501 20:10:14.375902 8139 consensus_queue.cc:237] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:14.379484 9368 tablet_bootstrap.cc:492] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:14.381835 9368 ts_tablet_manager.cc:1367] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.002s
I20240501 20:10:14.383631 9368 raft_consensus.cc:348] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:14.384608 9368 raft_consensus.cc:374] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:14.385231 9388 ts_tablet_manager.cc:902] T a20287c2cab44a62b28037a463d4b49b P ef0a4ada548446b3a531cb39a3c4e38d: Initiating tablet copy from peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:14.385423 9368 raft_consensus.cc:729] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:14.385783 9368 consensus_queue.cc:260] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:14.390837 9368 ts_tablet_manager.cc:1398] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 20:10:14.397233 9368 tablet_bootstrap.cc:492] T 024c925714cb419aaa8870a053a6ca2f P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:14.399050 9368 tablet_bootstrap.cc:654] T 024c925714cb419aaa8870a053a6ca2f P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:14.402957 9368 tablet_bootstrap.cc:492] T 024c925714cb419aaa8870a053a6ca2f P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:14.403669 9368 ts_tablet_manager.cc:1367] T 024c925714cb419aaa8870a053a6ca2f P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 20:10:14.404973 9368 raft_consensus.cc:348] T 024c925714cb419aaa8870a053a6ca2f P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:14.405148 9368 raft_consensus.cc:374] T 024c925714cb419aaa8870a053a6ca2f P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:14.406189 9368 raft_consensus.cc:729] T 024c925714cb419aaa8870a053a6ca2f P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:14.406911 9368 consensus_queue.cc:260] T 024c925714cb419aaa8870a053a6ca2f P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:14.409870 9368 ts_tablet_manager.cc:1398] T 024c925714cb419aaa8870a053a6ca2f P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:10:14.410207 9388 tablet_copy_client.cc:308] T a20287c2cab44a62b28037a463d4b49b P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.67:34595
I20240501 20:10:14.410167 4093 catalog_manager.cc:5484] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd reported cstate change: term changed from 0 to 1, leader changed from <none> to db0742ee3860468590eefef6ae6074fd (127.29.219.70). New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:14.411490 4519 tablet_copy_service.cc:140] P 3e974c03d3c546f9ab6c91e762a5426b: Received BeginTabletCopySession request for tablet a20287c2cab44a62b28037a463d4b49b from peer ef0a4ada548446b3a531cb39a3c4e38d ({username='slave'} at 127.29.219.69:60767)
I20240501 20:10:14.411768 4519 tablet_copy_service.cc:161] P 3e974c03d3c546f9ab6c91e762a5426b: Beginning new tablet copy session on tablet a20287c2cab44a62b28037a463d4b49b from peer ef0a4ada548446b3a531cb39a3c4e38d at {username='slave'} at 127.29.219.69:60767: session id = ef0a4ada548446b3a531cb39a3c4e38d-a20287c2cab44a62b28037a463d4b49b
I20240501 20:10:14.413503 8851 raft_consensus.cc:482] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:14.413678 8851 raft_consensus.cc:504] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:14.414450 8851 leader_election.cc:288] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:14.414894 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91193b2f5c1c4fc8ad4a443f73c75ed8" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:14.415130 7153 raft_consensus.cc:2454] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:10:14.415777 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91193b2f5c1c4fc8ad4a443f73c75ed8" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:14.416002 7022 raft_consensus.cc:2454] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:10:14.417523 4519 tablet_copy_source_session.cc:205] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:14.418143 7355 leader_election.cc:302] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:14.418442 8851 raft_consensus.cc:2781] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:14.418582 8851 raft_consensus.cc:482] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:14.418687 8851 raft_consensus.cc:3037] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:14.419731 9388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a20287c2cab44a62b28037a463d4b49b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:14.421983 8851 raft_consensus.cc:504] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:14.428185 9388 tablet_copy_client.cc:783] T a20287c2cab44a62b28037a463d4b49b P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:14.429199 9398 ts_tablet_manager.cc:902] T 23eaac7a50354947b6efa26ac5f01ae7 P bceeae1c79964acabdae8ff771bf0c97: Initiating tablet copy from peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:14.430224 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91193b2f5c1c4fc8ad4a443f73c75ed8" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:14.431672 8851 leader_election.cc:288] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Requested vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:14.432112 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91193b2f5c1c4fc8ad4a443f73c75ed8" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:14.432341 7153 raft_consensus.cc:3037] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:14.437193 7153 raft_consensus.cc:2454] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:14.437716 7353 leader_election.cc:302] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:14.438151 8851 raft_consensus.cc:2781] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:14.438931 7022 raft_consensus.cc:3037] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:14.439353 8851 raft_consensus.cc:686] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Becoming Leader. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Running, Role: LEADER
I20240501 20:10:14.439709 8851 consensus_queue.cc:237] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:14.444888 7022 raft_consensus.cc:2454] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:14.445875 9388 tablet_copy_client.cc:647] T a20287c2cab44a62b28037a463d4b49b P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:14.447273 9398 tablet_copy_client.cc:308] T 23eaac7a50354947b6efa26ac5f01ae7 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.68:45205
I20240501 20:10:14.448199 4651 tablet_copy_service.cc:140] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Received BeginTabletCopySession request for tablet 23eaac7a50354947b6efa26ac5f01ae7 from peer bceeae1c79964acabdae8ff771bf0c97 ({username='slave'} at 127.29.219.71:48361)
I20240501 20:10:14.448390 4651 tablet_copy_service.cc:161] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Beginning new tablet copy session on tablet 23eaac7a50354947b6efa26ac5f01ae7 from peer bceeae1c79964acabdae8ff771bf0c97 at {username='slave'} at 127.29.219.71:48361: session id = bceeae1c79964acabdae8ff771bf0c97-23eaac7a50354947b6efa26ac5f01ae7
I20240501 20:10:14.448752 4087 catalog_manager.cc:5484] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72). New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:14.449743 9388 tablet_copy_client.cc:523] T a20287c2cab44a62b28037a463d4b49b P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:14.451563 4651 tablet_copy_source_session.cc:205] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:14.453307 9398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23eaac7a50354947b6efa26ac5f01ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:14.454456 9388 tablet_bootstrap.cc:492] T a20287c2cab44a62b28037a463d4b49b P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:14.459998 9398 tablet_copy_client.cc:783] T 23eaac7a50354947b6efa26ac5f01ae7 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:14.460393 9398 tablet_copy_client.cc:647] T 23eaac7a50354947b6efa26ac5f01ae7 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:14.462450 9398 tablet_copy_client.cc:523] T 23eaac7a50354947b6efa26ac5f01ae7 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:14.465548 9398 tablet_bootstrap.cc:492] T 23eaac7a50354947b6efa26ac5f01ae7 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:14.479074 9020 raft_consensus.cc:482] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:14.479275 9020 raft_consensus.cc:504] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:14.480265 9020 leader_election.cc:288] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:14.481459 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "185ee8ce5e7f4617b0526b5f3fd64a29" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:10:14.481431 4497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "185ee8ce5e7f4617b0526b5f3fd64a29" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:14.481699 4365 raft_consensus.cc:2454] T 185ee8ce5e7f4617b0526b5f3fd64a29 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:14.481757 4497 raft_consensus.cc:2454] T 185ee8ce5e7f4617b0526b5f3fd64a29 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:14.482321 7222 leader_election.cc:302] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:14.482731 9020 raft_consensus.cc:2781] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:14.482851 9020 raft_consensus.cc:482] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:14.482966 9020 raft_consensus.cc:3037] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:14.485798 9020 raft_consensus.cc:504] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:14.486503 9020 leader_election.cc:288] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:14.487273 4497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "185ee8ce5e7f4617b0526b5f3fd64a29" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:10:14.487561 4497 raft_consensus.cc:3037] T 185ee8ce5e7f4617b0526b5f3fd64a29 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:14.491752 8851 raft_consensus.cc:482] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:14.491961 8851 raft_consensus.cc:504] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:14.492729 8851 leader_election.cc:288] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:14.494130 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "129322eab07149d2a6fb7eef96100e7a" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:14.494457 7287 raft_consensus.cc:2454] T 129322eab07149d2a6fb7eef96100e7a P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:10:14.495097 7354 leader_election.cc:302] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:14.495136 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "185ee8ce5e7f4617b0526b5f3fd64a29" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:10:14.495349 4365 raft_consensus.cc:3037] T 185ee8ce5e7f4617b0526b5f3fd64a29 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:14.495499 9403 raft_consensus.cc:2781] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:14.495602 9403 raft_consensus.cc:482] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:14.495721 9403 raft_consensus.cc:3037] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:14.498137 4365 raft_consensus.cc:2454] T 185ee8ce5e7f4617b0526b5f3fd64a29 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:14.498942 7222 leader_election.cc:302] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:14.499941 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "129322eab07149d2a6fb7eef96100e7a" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:14.500200 7022 raft_consensus.cc:2454] T 129322eab07149d2a6fb7eef96100e7a P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:10:14.501153 4497 raft_consensus.cc:2454] T 185ee8ce5e7f4617b0526b5f3fd64a29 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:14.502032 9403 raft_consensus.cc:504] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:14.502249 9131 raft_consensus.cc:2781] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:14.502418 9131 raft_consensus.cc:686] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Becoming Leader. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Running, Role: LEADER
I20240501 20:10:14.502727 9131 consensus_queue.cc:237] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:14.505693 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "129322eab07149d2a6fb7eef96100e7a" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:14.506035 7022 raft_consensus.cc:3037] T 129322eab07149d2a6fb7eef96100e7a P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:14.514806 9416 ts_tablet_manager.cc:902] T 91a2bf5ddc03431b963ab5da831fd340 P b9957d4086bc4a76b18ed9265a5739d5: Initiating tablet copy from peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:14.516610 7022 raft_consensus.cc:2454] T 129322eab07149d2a6fb7eef96100e7a P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:14.517375 7355 leader_election.cc:302] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:14.517753 8851 raft_consensus.cc:2781] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:14.521715 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "129322eab07149d2a6fb7eef96100e7a" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:14.522096 7287 raft_consensus.cc:3037] T 129322eab07149d2a6fb7eef96100e7a P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:14.522639 9398 tablet_bootstrap.cc:492] T 23eaac7a50354947b6efa26ac5f01ae7 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:14.523128 9398 tablet_bootstrap.cc:492] T 23eaac7a50354947b6efa26ac5f01ae7 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap complete.
I20240501 20:10:14.523417 9398 ts_tablet_manager.cc:1367] T 23eaac7a50354947b6efa26ac5f01ae7 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.058s user 0.042s sys 0.012s
I20240501 20:10:14.524255 9398 raft_consensus.cc:348] T 23eaac7a50354947b6efa26ac5f01ae7 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:14.524492 9398 raft_consensus.cc:729] T 23eaac7a50354947b6efa26ac5f01ae7 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: LEARNER
I20240501 20:10:14.524683 9398 consensus_queue.cc:260] T 23eaac7a50354947b6efa26ac5f01ae7 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:14.525424 9398 ts_tablet_manager.cc:1398] T 23eaac7a50354947b6efa26ac5f01ae7 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 20:10:14.529150 4651 tablet_copy_service.cc:342] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Request end of tablet copy session bceeae1c79964acabdae8ff771bf0c97-23eaac7a50354947b6efa26ac5f01ae7 received from {username='slave'} at 127.29.219.71:48361
I20240501 20:10:14.529309 4651 tablet_copy_service.cc:433] P 1d34e18a1d394d3f91c9f50d1c7bea4e: ending tablet copy session bceeae1c79964acabdae8ff771bf0c97-23eaac7a50354947b6efa26ac5f01ae7 on tablet 23eaac7a50354947b6efa26ac5f01ae7 with peer bceeae1c79964acabdae8ff771bf0c97
I20240501 20:10:14.529078 4087 catalog_manager.cc:5484] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71). New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:14.531792 9403 leader_election.cc:288] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:14.536991 9388 tablet_bootstrap.cc:492] T a20287c2cab44a62b28037a463d4b49b P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:14.537569 9388 tablet_bootstrap.cc:492] T a20287c2cab44a62b28037a463d4b49b P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap complete.
I20240501 20:10:14.537889 9388 ts_tablet_manager.cc:1367] T a20287c2cab44a62b28037a463d4b49b P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.084s user 0.047s sys 0.005s
I20240501 20:10:14.538789 9388 raft_consensus.cc:348] T a20287c2cab44a62b28037a463d4b49b P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:14.539512 9388 raft_consensus.cc:729] T a20287c2cab44a62b28037a463d4b49b P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: LEARNER
I20240501 20:10:14.540002 7287 raft_consensus.cc:2454] T 129322eab07149d2a6fb7eef96100e7a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:14.540030 9388 consensus_queue.cc:260] T a20287c2cab44a62b28037a463d4b49b P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:14.540828 9388 ts_tablet_manager.cc:1398] T a20287c2cab44a62b28037a463d4b49b P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:14.547420 8851 raft_consensus.cc:686] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Becoming Leader. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Running, Role: LEADER
I20240501 20:10:14.547791 8851 consensus_queue.cc:237] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:14.559347 4519 tablet_copy_service.cc:342] P 3e974c03d3c546f9ab6c91e762a5426b: Request end of tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-a20287c2cab44a62b28037a463d4b49b received from {username='slave'} at 127.29.219.69:60767
I20240501 20:10:14.559617 4519 tablet_copy_service.cc:433] P 3e974c03d3c546f9ab6c91e762a5426b: ending tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-a20287c2cab44a62b28037a463d4b49b on tablet a20287c2cab44a62b28037a463d4b49b with peer ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:14.563943 4087 catalog_manager.cc:5484] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72). New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:14.565989 9416 tablet_copy_client.cc:308] T 91a2bf5ddc03431b963ab5da831fd340 P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.66:40689
I20240501 20:10:14.566733 4388 tablet_copy_service.cc:140] P 10b47782558c41ee91f5e164a372049a: Received BeginTabletCopySession request for tablet 91a2bf5ddc03431b963ab5da831fd340 from peer b9957d4086bc4a76b18ed9265a5739d5 ({username='slave'} at 127.29.219.72:44421)
I20240501 20:10:14.566936 4388 tablet_copy_service.cc:161] P 10b47782558c41ee91f5e164a372049a: Beginning new tablet copy session on tablet 91a2bf5ddc03431b963ab5da831fd340 from peer b9957d4086bc4a76b18ed9265a5739d5 at {username='slave'} at 127.29.219.72:44421: session id = b9957d4086bc4a76b18ed9265a5739d5-91a2bf5ddc03431b963ab5da831fd340
I20240501 20:10:14.569739 4388 tablet_copy_source_session.cc:205] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:14.572160 9416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a2bf5ddc03431b963ab5da831fd340. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:14.578812 9416 tablet_copy_client.cc:783] T 91a2bf5ddc03431b963ab5da831fd340 P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:14.579151 9416 tablet_copy_client.cc:647] T 91a2bf5ddc03431b963ab5da831fd340 P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:14.581197 9416 tablet_copy_client.cc:523] T 91a2bf5ddc03431b963ab5da831fd340 P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:14.584244 9416 tablet_bootstrap.cc:492] T 91a2bf5ddc03431b963ab5da831fd340 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:14.601141 4087 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:14.634441 7022 raft_consensus.cc:1261] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:14.636091 8851 consensus_queue.cc:1035] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:14.639178 7155 raft_consensus.cc:1261] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:14.640169 9419 consensus_queue.cc:1035] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:14.640370 9437 ts_tablet_manager.cc:902] T a8eefa22b6e74b92b9fef036dea46e3b P db0742ee3860468590eefef6ae6074fd: Initiating tablet copy from peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:14.650876 4087 catalog_manager.cc:5576] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321) does not have the latest schema for tablet 91193b2f5c1c4fc8ad4a443f73c75ed8 (table rebalancer_dynamic_table [id=ea651d4353a24bc3a854da0192253c56]). Expected version 1 got 0
I20240501 20:10:14.664554 9444 tablet.cc:1722] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:14.665285 9442 tablet.cc:1722] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:14.665728 4087 catalog_manager.cc:5576] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237) does not have the latest schema for tablet 91193b2f5c1c4fc8ad4a443f73c75ed8 (table rebalancer_dynamic_table [id=ea651d4353a24bc3a854da0192253c56]). Expected version 1 got 0
I20240501 20:10:14.669260 4087 catalog_manager.cc:5576] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321) does not have the latest schema for tablet 91193b2f5c1c4fc8ad4a443f73c75ed8 (table rebalancer_dynamic_table [id=ea651d4353a24bc3a854da0192253c56]). Expected version 1 got 0
I20240501 20:10:14.670025 9442 tablet.cc:1620] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:14.672049 4087 catalog_manager.cc:5576] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817) does not have the latest schema for tablet 91193b2f5c1c4fc8ad4a443f73c75ed8 (table rebalancer_dynamic_table [id=ea651d4353a24bc3a854da0192253c56]). Expected version 1 got 0
I20240501 20:10:14.666996 9444 tablet.cc:1620] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:10:14.675724 9448 tablet.cc:1722] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:14.677332 9020 raft_consensus.cc:482] T 024c925714cb419aaa8870a053a6ca2f P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:14.677539 9020 raft_consensus.cc:504] T 024c925714cb419aaa8870a053a6ca2f P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:14.678316 9020 leader_election.cc:288] T 024c925714cb419aaa8870a053a6ca2f P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:14.678728 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024c925714cb419aaa8870a053a6ca2f" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:14.679651 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024c925714cb419aaa8870a053a6ca2f" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:14.679926 4234 raft_consensus.cc:2379] T 024c925714cb419aaa8870a053a6ca2f P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bceeae1c79964acabdae8ff771bf0c97 in current term 1: Already voted for candidate db0742ee3860468590eefef6ae6074fd in this term.
I20240501 20:10:14.680415 7222 leader_election.cc:302] T 024c925714cb419aaa8870a053a6ca2f P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97; no voters: db0742ee3860468590eefef6ae6074fd, dca57823003e4729b769ca12367d7530
I20240501 20:10:14.680990 9020 raft_consensus.cc:3037] T 024c925714cb419aaa8870a053a6ca2f P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:14.683684 9020 raft_consensus.cc:2726] T 024c925714cb419aaa8870a053a6ca2f P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:14.683979 4093 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=ea651d4353a24bc3a854da0192253c56] alter complete (version 1)
I20240501 20:10:14.687681 4087 catalog_manager.cc:5576] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237) does not have the latest schema for tablet 91193b2f5c1c4fc8ad4a443f73c75ed8 (table rebalancer_dynamic_table [id=ea651d4353a24bc3a854da0192253c56]). Expected version 1 got 0
I20240501 20:10:14.689466 9448 tablet.cc:1620] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:14.699375 9437 tablet_copy_client.cc:308] T a8eefa22b6e74b92b9fef036dea46e3b P db0742ee3860468590eefef6ae6074fd: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.65:42941
I20240501 20:10:14.700230 4254 tablet_copy_service.cc:140] P dca57823003e4729b769ca12367d7530: Received BeginTabletCopySession request for tablet a8eefa22b6e74b92b9fef036dea46e3b from peer db0742ee3860468590eefef6ae6074fd ({username='slave'} at 127.29.219.70:47215)
I20240501 20:10:14.700387 4254 tablet_copy_service.cc:161] P dca57823003e4729b769ca12367d7530: Beginning new tablet copy session on tablet a8eefa22b6e74b92b9fef036dea46e3b from peer db0742ee3860468590eefef6ae6074fd at {username='slave'} at 127.29.219.70:47215: session id = db0742ee3860468590eefef6ae6074fd-a8eefa22b6e74b92b9fef036dea46e3b
I20240501 20:10:14.703548 4254 tablet_copy_source_session.cc:205] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:14.706158 9437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8eefa22b6e74b92b9fef036dea46e3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:14.714656 9437 tablet_copy_client.cc:783] T a8eefa22b6e74b92b9fef036dea46e3b P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:14.715195 9437 tablet_copy_client.cc:647] T a8eefa22b6e74b92b9fef036dea46e3b P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:14.717197 9437 tablet_copy_client.cc:523] T a8eefa22b6e74b92b9fef036dea46e3b P db0742ee3860468590eefef6ae6074fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:14.721738 9450 ts_tablet_manager.cc:902] T f82a24aabb2b4386b5916653a97e548c P 10b47782558c41ee91f5e164a372049a: Initiating tablet copy from peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:14.739346 9437 tablet_bootstrap.cc:492] T a8eefa22b6e74b92b9fef036dea46e3b P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:14.767882 7021 raft_consensus.cc:1203] T a20287c2cab44a62b28037a463d4b49b P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:14.777817 9416 tablet_bootstrap.cc:492] T 91a2bf5ddc03431b963ab5da831fd340 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:14.778321 9416 tablet_bootstrap.cc:492] T 91a2bf5ddc03431b963ab5da831fd340 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap complete.
I20240501 20:10:14.780826 9416 ts_tablet_manager.cc:1367] T 91a2bf5ddc03431b963ab5da831fd340 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.197s user 0.062s sys 0.003s
I20240501 20:10:14.781711 9416 raft_consensus.cc:348] T 91a2bf5ddc03431b963ab5da831fd340 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:14.781991 9416 raft_consensus.cc:729] T 91a2bf5ddc03431b963ab5da831fd340 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: LEARNER
I20240501 20:10:14.782222 9416 consensus_queue.cc:260] T 91a2bf5ddc03431b963ab5da831fd340 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:14.782812 9416 ts_tablet_manager.cc:1398] T 91a2bf5ddc03431b963ab5da831fd340 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:14.786557 4388 tablet_copy_service.cc:342] P 10b47782558c41ee91f5e164a372049a: Request end of tablet copy session b9957d4086bc4a76b18ed9265a5739d5-91a2bf5ddc03431b963ab5da831fd340 received from {username='slave'} at 127.29.219.72:44421
I20240501 20:10:14.786711 4388 tablet_copy_service.cc:433] P 10b47782558c41ee91f5e164a372049a: ending tablet copy session b9957d4086bc4a76b18ed9265a5739d5-91a2bf5ddc03431b963ab5da831fd340 on tablet 91a2bf5ddc03431b963ab5da831fd340 with peer b9957d4086bc4a76b18ed9265a5739d5
I20240501 20:10:14.787542 4093 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:14.813419 8139 consensus_queue.cc:1035] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:14.818329 9442 tablet.cc:1722] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:14.819043 9442 tablet.cc:1620] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:14.822504 9444 tablet.cc:1722] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:14.823190 9444 tablet.cc:1620] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:10:14.831527 9448 tablet.cc:1722] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:14.832067 9448 tablet.cc:1620] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:14.844751 4091 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=ea651d4353a24bc3a854da0192253c56] alter complete (version 2)
I20240501 20:10:14.847221 9033 consensus_queue.cc:1035] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:14.877676 7285 raft_consensus.cc:1203] T 23eaac7a50354947b6efa26ac5f01ae7 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:14.895011 9450 tablet_copy_client.cc:308] T f82a24aabb2b4386b5916653a97e548c P 10b47782558c41ee91f5e164a372049a: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.71:39447
I20240501 20:10:14.896243 7308 tablet_copy_service.cc:140] P bceeae1c79964acabdae8ff771bf0c97: Received BeginTabletCopySession request for tablet f82a24aabb2b4386b5916653a97e548c from peer 10b47782558c41ee91f5e164a372049a ({username='slave'} at 127.29.219.66:32801)
I20240501 20:10:14.896428 7308 tablet_copy_service.cc:161] P bceeae1c79964acabdae8ff771bf0c97: Beginning new tablet copy session on tablet f82a24aabb2b4386b5916653a97e548c from peer 10b47782558c41ee91f5e164a372049a at {username='slave'} at 127.29.219.66:32801: session id = 10b47782558c41ee91f5e164a372049a-f82a24aabb2b4386b5916653a97e548c
I20240501 20:10:14.899397 7308 tablet_copy_source_session.cc:205] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:14.901055 9450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f82a24aabb2b4386b5916653a97e548c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:14.908375 9450 tablet_copy_client.cc:783] T f82a24aabb2b4386b5916653a97e548c P 10b47782558c41ee91f5e164a372049a: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:14.908761 9450 tablet_copy_client.cc:647] T f82a24aabb2b4386b5916653a97e548c P 10b47782558c41ee91f5e164a372049a: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:14.912465 9450 tablet_copy_client.cc:523] T f82a24aabb2b4386b5916653a97e548c P 10b47782558c41ee91f5e164a372049a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:14.916347 9450 tablet_bootstrap.cc:492] T f82a24aabb2b4386b5916653a97e548c P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:14.937052 9450 tablet_bootstrap.cc:492] T f82a24aabb2b4386b5916653a97e548c P 10b47782558c41ee91f5e164a372049a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:14.937772 9450 tablet_bootstrap.cc:492] T f82a24aabb2b4386b5916653a97e548c P 10b47782558c41ee91f5e164a372049a: Bootstrap complete.
I20240501 20:10:14.938115 9450 ts_tablet_manager.cc:1367] T f82a24aabb2b4386b5916653a97e548c P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.004s
I20240501 20:10:14.939185 9450 raft_consensus.cc:348] T f82a24aabb2b4386b5916653a97e548c P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:14.939574 9450 raft_consensus.cc:729] T f82a24aabb2b4386b5916653a97e548c P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: LEARNER
I20240501 20:10:14.939898 9450 consensus_queue.cc:260] T f82a24aabb2b4386b5916653a97e548c P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:14.941913 9450 ts_tablet_manager.cc:1398] T f82a24aabb2b4386b5916653a97e548c P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:10:14.942780 7308 tablet_copy_service.cc:342] P bceeae1c79964acabdae8ff771bf0c97: Request end of tablet copy session 10b47782558c41ee91f5e164a372049a-f82a24aabb2b4386b5916653a97e548c received from {username='slave'} at 127.29.219.66:32801
I20240501 20:10:14.942947 7308 tablet_copy_service.cc:433] P bceeae1c79964acabdae8ff771bf0c97: ending tablet copy session 10b47782558c41ee91f5e164a372049a-f82a24aabb2b4386b5916653a97e548c on tablet f82a24aabb2b4386b5916653a97e548c with peer 10b47782558c41ee91f5e164a372049a
I20240501 20:10:14.952033 9437 tablet_bootstrap.cc:492] T a8eefa22b6e74b92b9fef036dea46e3b P db0742ee3860468590eefef6ae6074fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:14.952147 4088 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:14.952507 9437 tablet_bootstrap.cc:492] T a8eefa22b6e74b92b9fef036dea46e3b P db0742ee3860468590eefef6ae6074fd: Bootstrap complete.
I20240501 20:10:14.952767 9437 ts_tablet_manager.cc:1367] T a8eefa22b6e74b92b9fef036dea46e3b P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.214s user 0.048s sys 0.003s
I20240501 20:10:14.953519 9437 raft_consensus.cc:348] T a8eefa22b6e74b92b9fef036dea46e3b P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:14.953735 9437 raft_consensus.cc:729] T a8eefa22b6e74b92b9fef036dea46e3b P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: LEARNER
I20240501 20:10:14.953939 9437 consensus_queue.cc:260] T a8eefa22b6e74b92b9fef036dea46e3b P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:14.954404 9437 ts_tablet_manager.cc:1398] T a8eefa22b6e74b92b9fef036dea46e3b P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:14.955199 4254 tablet_copy_service.cc:342] P dca57823003e4729b769ca12367d7530: Request end of tablet copy session db0742ee3860468590eefef6ae6074fd-a8eefa22b6e74b92b9fef036dea46e3b received from {username='slave'} at 127.29.219.70:47215
I20240501 20:10:14.955346 4254 tablet_copy_service.cc:433] P dca57823003e4729b769ca12367d7530: ending tablet copy session db0742ee3860468590eefef6ae6074fd-a8eefa22b6e74b92b9fef036dea46e3b on tablet a8eefa22b6e74b92b9fef036dea46e3b with peer db0742ee3860468590eefef6ae6074fd
I20240501 20:10:14.969844 9442 tablet.cc:1722] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:14.970775 9448 tablet.cc:1722] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:14.971189 9444 tablet.cc:1722] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:14.986200 4091 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table_new_name [id=ea651d4353a24bc3a854da0192253c56] alter complete (version 3)
I20240501 20:10:14.997758 4497 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4f99f5d9d4bf44c39188c8b5d5362b98"
dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35048
I20240501 20:10:14.998015 4497 raft_consensus.cc:595] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Received request to transfer leadership
I20240501 20:10:15.002440 9419 consensus_queue.cc:1035] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:15.008837 9403 consensus_queue.cc:1035] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:15.039755 4497 consensus_queue.cc:237] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:15.042045 4365 raft_consensus.cc:1261] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:15.042192 4627 raft_consensus.cc:1261] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:15.042605 4433 consensus_peers.cc:485] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b -> Peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Couldn't send request to peer bceeae1c79964acabdae8ff771bf0c97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65b0048212a745c08ab24ead346bf74e. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:15.042697 9169 consensus_queue.cc:1035] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:15.043116 9319 consensus_queue.cc:1035] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:15.047102 9319 raft_consensus.cc:2932] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:15.047806 4627 raft_consensus.cc:2932] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:15.051620 4365 raft_consensus.cc:2932] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:15.054212 4093 catalog_manager.cc:5484] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:15.054956 7419 raft_consensus.cc:1203] T 91a2bf5ddc03431b963ab5da831fd340 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:15.086025 9020 consensus_queue.cc:1035] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:15.089630 4087 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:15.089815 4087 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:15.093269 4087 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 91193b2f5c1c4fc8ad4a443f73c75ed8
I20240501 20:10:15.094449 7401 tablet_service.cc:1514] Processing DeleteTablet for tablet 91193b2f5c1c4fc8ad4a443f73c75ed8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:15 UTC) from {username='slave'} at 127.0.0.1:40182
I20240501 20:10:15.095146 9508 tablet_replica.cc:330] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5: stopping tablet replica
I20240501 20:10:15.095466 9508 raft_consensus.cc:2229] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:15.095947 9508 raft_consensus.cc:2258] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:15.097115 7133 tablet_service.cc:1514] Processing DeleteTablet for tablet 91193b2f5c1c4fc8ad4a443f73c75ed8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:15 UTC) from {username='slave'} at 127.0.0.1:52036
I20240501 20:10:15.097294 9508 ts_tablet_manager.cc:1875] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:15.097859 7002 tablet_service.cc:1514] Processing DeleteTablet for tablet 91193b2f5c1c4fc8ad4a443f73c75ed8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:15 UTC) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:15.098711 9510 tablet_replica.cc:330] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P ef0a4ada548446b3a531cb39a3c4e38d: stopping tablet replica
I20240501 20:10:15.099063 9510 raft_consensus.cc:2229] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:15.099328 9510 raft_consensus.cc:2258] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:15.100595 9510 ts_tablet_manager.cc:1875] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:15.102788 7153 raft_consensus.cc:1203] T a8eefa22b6e74b92b9fef036dea46e3b P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:15.111141 9510 ts_tablet_manager.cc:1888] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P ef0a4ada548446b3a531cb39a3c4e38d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:15.111433 9510 log.cc:1196] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/wals/91193b2f5c1c4fc8ad4a443f73c75ed8
I20240501 20:10:15.112162 9510 ts_tablet_manager.cc:1909] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting consensus metadata
I20240501 20:10:15.113865 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet 91193b2f5c1c4fc8ad4a443f73c75ed8 (table rebalancer_dynamic_table_new_name [id=ea651d4353a24bc3a854da0192253c56]) successfully deleted
I20240501 20:10:15.121973 9508 ts_tablet_manager.cc:1888] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:15.123373 9509 tablet_replica.cc:330] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P db0742ee3860468590eefef6ae6074fd: stopping tablet replica
I20240501 20:10:15.123589 9508 log.cc:1196] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/wals/91193b2f5c1c4fc8ad4a443f73c75ed8
I20240501 20:10:15.123914 9509 raft_consensus.cc:2229] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:15.124315 9509 raft_consensus.cc:2258] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:15.124436 9508 ts_tablet_manager.cc:1909] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P b9957d4086bc4a76b18ed9265a5739d5: Deleting consensus metadata
I20240501 20:10:15.125869 9513 consensus_queue.cc:1035] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:15.126463 4079 catalog_manager.cc:4830] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): tablet 91193b2f5c1c4fc8ad4a443f73c75ed8 (table rebalancer_dynamic_table_new_name [id=ea651d4353a24bc3a854da0192253c56]) successfully deleted
I20240501 20:10:15.130194 4087 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:15.131590 9509 ts_tablet_manager.cc:1875] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P db0742ee3860468590eefef6ae6074fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:15.145972 7267 tablet_service.cc:1467] Processing CreateTablet for tablet 7ecf2962444644cc8c5bdf5caf95c8c8 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=affa3c54d7ce469c819876e9e5aedb60]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:15.146597 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ecf2962444644cc8c5bdf5caf95c8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:15.167508 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 7ecf2962444644cc8c5bdf5caf95c8c8 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=affa3c54d7ce469c819876e9e5aedb60]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:15.167587 7002 tablet_service.cc:1467] Processing CreateTablet for tablet 7ecf2962444644cc8c5bdf5caf95c8c8 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=affa3c54d7ce469c819876e9e5aedb60]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:15.168151 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ecf2962444644cc8c5bdf5caf95c8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:15.168146 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ecf2962444644cc8c5bdf5caf95c8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:15.166534 9319 raft_consensus.cc:979] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b: : Instructing follower b9957d4086bc4a76b18ed9265a5739d5 to start an election
I20240501 20:10:15.170977 9319 raft_consensus.cc:1067] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Signalling peer b9957d4086bc4a76b18ed9265a5739d5 to start an election
I20240501 20:10:15.172091 7419 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f99f5d9d4bf44c39188c8b5d5362b98"
dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
from {username='slave'} at 127.29.219.67:43167
I20240501 20:10:15.172415 7419 raft_consensus.cc:482] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:10:15.172550 7419 raft_consensus.cc:3037] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:15.177024 7133 tablet_service.cc:1467] Processing CreateTablet for tablet 79af181b5f014a40a4c123b47a720dec (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=affa3c54d7ce469c819876e9e5aedb60]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:15.177521 7135 tablet_service.cc:1467] Processing CreateTablet for tablet 5a7b2f01b5684df5800987591a390a17 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=affa3c54d7ce469c819876e9e5aedb60]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:15.178236 4346 tablet_service.cc:1467] Processing CreateTablet for tablet 5a7b2f01b5684df5800987591a390a17 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=affa3c54d7ce469c819876e9e5aedb60]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:15.178260 4347 tablet_service.cc:1467] Processing CreateTablet for tablet 79af181b5f014a40a4c123b47a720dec (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=affa3c54d7ce469c819876e9e5aedb60]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:15.178686 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a7b2f01b5684df5800987591a390a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:15.178961 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79af181b5f014a40a4c123b47a720dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:15.179085 7000 tablet_service.cc:1467] Processing CreateTablet for tablet 5a7b2f01b5684df5800987591a390a17 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=affa3c54d7ce469c819876e9e5aedb60]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:15.179505 7000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a7b2f01b5684df5800987591a390a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:15.179792 7001 tablet_service.cc:1467] Processing CreateTablet for tablet 79af181b5f014a40a4c123b47a720dec (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=affa3c54d7ce469c819876e9e5aedb60]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:15.180192 7001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79af181b5f014a40a4c123b47a720dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:15.177521 7133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79af181b5f014a40a4c123b47a720dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:15.186800 7135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a7b2f01b5684df5800987591a390a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:15.195684 4087 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:15.200335 7419 raft_consensus.cc:504] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:15.202883 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f99f5d9d4bf44c39188c8b5d5362b98" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:10:15.202941 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f99f5d9d4bf44c39188c8b5d5362b98" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:10:15.203130 4234 raft_consensus.cc:3037] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:15.203162 4627 raft_consensus.cc:3037] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:15.204159 9519 tablet_bootstrap.cc:492] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:15.205896 4234 raft_consensus.cc:2454] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 2.
I20240501 20:10:15.206332 4627 raft_consensus.cc:2454] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 2.
I20240501 20:10:15.206573 7419 leader_election.cc:288] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:15.206852 7356 leader_election.cc:302] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, b9957d4086bc4a76b18ed9265a5739d5, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:15.207495 9419 raft_consensus.cc:2781] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:15.207664 9419 raft_consensus.cc:686] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 LEADER]: Becoming Leader. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Running, Role: LEADER
I20240501 20:10:15.208046 9419 consensus_queue.cc:237] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:15.211462 9519 tablet_bootstrap.cc:654] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:15.212663 4497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f99f5d9d4bf44c39188c8b5d5362b98" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:10:15.212967 4497 raft_consensus.cc:3032] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:10:15.213083 4497 raft_consensus.cc:729] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:10:15.213809 4497 consensus_queue.cc:260] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:15.214361 4497 raft_consensus.cc:3037] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:15.214762 9521 tablet_bootstrap.cc:492] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:15.222457 9509 ts_tablet_manager.cc:1888] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P db0742ee3860468590eefef6ae6074fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:15.224120 9519 tablet_bootstrap.cc:492] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:15.224871 9519 ts_tablet_manager.cc:1367] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240501 20:10:15.227603 9519 raft_consensus.cc:348] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:15.229086 9509 log.cc:1196] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P db0742ee3860468590eefef6ae6074fd: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/wals/91193b2f5c1c4fc8ad4a443f73c75ed8
I20240501 20:10:15.229631 9519 raft_consensus.cc:374] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:15.229909 9519 raft_consensus.cc:729] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:15.230016 9509 ts_tablet_manager.cc:1909] T 91193b2f5c1c4fc8ad4a443f73c75ed8 P db0742ee3860468590eefef6ae6074fd: Deleting consensus metadata
I20240501 20:10:15.231251 9519 consensus_queue.cc:260] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:15.233271 9519 ts_tablet_manager.cc:1398] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 20:10:15.238456 4076 catalog_manager.cc:4830] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): tablet 91193b2f5c1c4fc8ad4a443f73c75ed8 (table rebalancer_dynamic_table_new_name [id=ea651d4353a24bc3a854da0192253c56]) successfully deleted
I20240501 20:10:15.238831 9525 tablet_bootstrap.cc:492] T 5a7b2f01b5684df5800987591a390a17 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:15.240825 9521 tablet_bootstrap.cc:654] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:15.240835 4367 raft_consensus.cc:1203] T f82a24aabb2b4386b5916653a97e548c P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:15.246222 7002 tablet_service.cc:1467] Processing CreateTablet for tablet ae106dc4a9d646cf8c6947e6895e03cd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=04857c94591e4e16a9a3f7f9117b25ae]), partition=
I20240501 20:10:15.247113 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae106dc4a9d646cf8c6947e6895e03cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:15.253891 4497 raft_consensus.cc:2454] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 2.
I20240501 20:10:15.255288 9525 tablet_bootstrap.cc:654] T 5a7b2f01b5684df5800987591a390a17 P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:15.260618 4215 tablet_service.cc:1467] Processing CreateTablet for tablet ae106dc4a9d646cf8c6947e6895e03cd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=04857c94591e4e16a9a3f7f9117b25ae]), partition=
I20240501 20:10:15.261013 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae106dc4a9d646cf8c6947e6895e03cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:15.263319 7135 tablet_service.cc:1467] Processing CreateTablet for tablet ae106dc4a9d646cf8c6947e6895e03cd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=04857c94591e4e16a9a3f7f9117b25ae]), partition=
I20240501 20:10:15.263773 7135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae106dc4a9d646cf8c6947e6895e03cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:15.287199 9529 tablet_bootstrap.cc:492] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:15.289547 9529 tablet_bootstrap.cc:654] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:15.291864 9521 tablet_bootstrap.cc:492] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:15.292153 9521 ts_tablet_manager.cc:1367] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.078s user 0.002s sys 0.005s
I20240501 20:10:15.296257 9520 tablet_bootstrap.cc:492] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:15.297091 9525 tablet_bootstrap.cc:492] T 5a7b2f01b5684df5800987591a390a17 P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:15.297363 9525 ts_tablet_manager.cc:1367] T 5a7b2f01b5684df5800987591a390a17 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.061s user 0.000s sys 0.007s
I20240501 20:10:15.298833 9525 raft_consensus.cc:348] T 5a7b2f01b5684df5800987591a390a17 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:15.299131 9525 raft_consensus.cc:374] T 5a7b2f01b5684df5800987591a390a17 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:15.299361 9525 raft_consensus.cc:729] T 5a7b2f01b5684df5800987591a390a17 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:15.299736 9525 consensus_queue.cc:260] T 5a7b2f01b5684df5800987591a390a17 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:15.307749 9525 ts_tablet_manager.cc:1398] T 5a7b2f01b5684df5800987591a390a17 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 20:10:15.309918 4087 catalog_manager.cc:5484] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: term changed from 1 to 2, leader changed from 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) to b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72). New cstate: current_term: 2 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:15.314496 9522 tablet_bootstrap.cc:492] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:15.316670 9525 tablet_bootstrap.cc:492] T 79af181b5f014a40a4c123b47a720dec P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:15.321162 9525 tablet_bootstrap.cc:654] T 79af181b5f014a40a4c123b47a720dec P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:15.325281 9529 tablet_bootstrap.cc:492] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:15.293272 9521 raft_consensus.cc:348] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:15.325536 9529 ts_tablet_manager.cc:1367] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20240501 20:10:15.325603 9521 raft_consensus.cc:374] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:15.326417 9529 raft_consensus.cc:348] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:15.326570 9521 raft_consensus.cc:729] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:15.327026 9521 consensus_queue.cc:260] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:15.327172 9529 raft_consensus.cc:374] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:15.327239 9529 raft_consensus.cc:729] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:15.327476 9529 consensus_queue.cc:260] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:15.329172 9529 ts_tablet_manager.cc:1398] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:15.333186 9419 raft_consensus.cc:482] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:15.333526 9419 raft_consensus.cc:504] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:15.335933 9419 leader_election.cc:288] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:15.336488 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ecf2962444644cc8c5bdf5caf95c8c8" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
W20240501 20:10:15.337361 7354 leader_election.cc:341] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:15.339336 7021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ecf2962444644cc8c5bdf5caf95c8c8" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:15.345439 9522 tablet_bootstrap.cc:654] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
W20240501 20:10:15.346598 7355 leader_election.cc:341] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:15.346825 7355 leader_election.cc:302] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5; no voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:15.347430 9419 raft_consensus.cc:2726] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:15.350342 8790 raft_consensus.cc:482] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:15.350499 8790 raft_consensus.cc:504] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:15.350700 9520 tablet_bootstrap.cc:654] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:15.351269 8790 leader_election.cc:288] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:15.351882 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae106dc4a9d646cf8c6947e6895e03cd" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:15.352373 7021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae106dc4a9d646cf8c6947e6895e03cd" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
W20240501 20:10:15.352675 4167 leader_election.cc:341] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:10:15.353547 4169 leader_election.cc:341] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:15.353678 4169 leader_election.cc:302] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dca57823003e4729b769ca12367d7530; no voters: db0742ee3860468590eefef6ae6074fd, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:15.354482 8785 raft_consensus.cc:2726] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:15.355029 9521 ts_tablet_manager.cc:1398] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.063s user 0.003s sys 0.000s
I20240501 20:10:15.365474 9521 tablet_bootstrap.cc:492] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:15.368043 9521 tablet_bootstrap.cc:654] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:15.375106 9525 tablet_bootstrap.cc:492] T 79af181b5f014a40a4c123b47a720dec P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:15.375341 9525 ts_tablet_manager.cc:1367] T 79af181b5f014a40a4c123b47a720dec P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.005s
I20240501 20:10:15.376243 9525 raft_consensus.cc:348] T 79af181b5f014a40a4c123b47a720dec P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:15.376504 9525 raft_consensus.cc:374] T 79af181b5f014a40a4c123b47a720dec P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:15.376590 9525 raft_consensus.cc:729] T 79af181b5f014a40a4c123b47a720dec P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:15.376883 9525 consensus_queue.cc:260] T 79af181b5f014a40a4c123b47a720dec P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:15.377312 9169 raft_consensus.cc:1050] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b: attempting to promote NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d to VOTER
I20240501 20:10:15.377434 9525 ts_tablet_manager.cc:1398] T 79af181b5f014a40a4c123b47a720dec P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:15.377792 9525 tablet_bootstrap.cc:492] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:15.378196 9169 consensus_queue.cc:237] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:15.385531 9525 tablet_bootstrap.cc:654] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:15.394073 7021 raft_consensus.cc:1261] T a20287c2cab44a62b28037a463d4b49b P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:15.394563 4627 raft_consensus.cc:1261] T a20287c2cab44a62b28037a463d4b49b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:15.395157 9457 consensus_queue.cc:1035] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:15.395742 9457 consensus_queue.cc:1035] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:15.405017 4365 raft_consensus.cc:1261] T a20287c2cab44a62b28037a463d4b49b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:15.405692 9169 consensus_queue.cc:1035] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:15.406113 9525 tablet_bootstrap.cc:492] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:15.406311 9525 ts_tablet_manager.cc:1367] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.006s
I20240501 20:10:15.407688 9525 raft_consensus.cc:348] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:15.408021 9525 raft_consensus.cc:374] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:15.408164 9525 raft_consensus.cc:729] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:15.408530 9525 consensus_queue.cc:260] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:15.409432 9525 ts_tablet_manager.cc:1398] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240501 20:10:15.414446 9520 tablet_bootstrap.cc:492] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:15.414680 9520 ts_tablet_manager.cc:1367] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.119s user 0.005s sys 0.000s
I20240501 20:10:15.417656 9520 raft_consensus.cc:348] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:15.419684 9522 tablet_bootstrap.cc:492] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:10:15.421926 9522 ts_tablet_manager.cc:1367] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.108s user 0.002s sys 0.003s
I20240501 20:10:15.433460 9520 raft_consensus.cc:374] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:15.433837 9520 raft_consensus.cc:729] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:15.434243 9520 consensus_queue.cc:260] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:15.435431 9520 ts_tablet_manager.cc:1398] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240501 20:10:15.437844 9541 raft_consensus.cc:2932] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:15.442960 9522 raft_consensus.cc:348] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:15.443372 9522 raft_consensus.cc:374] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:15.443507 9522 raft_consensus.cc:729] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:10:15.443799 9522 consensus_queue.cc:260] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:15.454857 9522 ts_tablet_manager.cc:1398] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20240501 20:10:15.455392 4627 raft_consensus.cc:2932] T a20287c2cab44a62b28037a463d4b49b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:15.462074 9521 tablet_bootstrap.cc:492] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:15.462479 9521 ts_tablet_manager.cc:1367] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.097s user 0.005s sys 0.000s
I20240501 20:10:15.463835 8604 raft_consensus.cc:482] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:15.464041 8604 raft_consensus.cc:504] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:15.465061 8604 leader_election.cc:288] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:15.465132 4365 raft_consensus.cc:2932] T a20287c2cab44a62b28037a463d4b49b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:15.466701 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a7b2f01b5684df5800987591a390a17" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:15.466984 7155 raft_consensus.cc:2454] T 5a7b2f01b5684df5800987591a390a17 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:10:15.467720 4299 leader_election.cc:302] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:15.468592 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a7b2f01b5684df5800987591a390a17" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:15.469112 8604 raft_consensus.cc:2781] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:15.469250 8604 raft_consensus.cc:482] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:15.469352 8604 raft_consensus.cc:3037] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:15.473343 8604 raft_consensus.cc:504] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:15.473230 4087 catalog_manager.cc:5484] T a20287c2cab44a62b28037a463d4b49b P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:15.474812 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a7b2f01b5684df5800987591a390a17" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:15.474897 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a7b2f01b5684df5800987591a390a17" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:15.475158 7155 raft_consensus.cc:3037] T 5a7b2f01b5684df5800987591a390a17 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:15.482533 9332 raft_consensus.cc:1050] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e: attempting to promote NON_VOTER bceeae1c79964acabdae8ff771bf0c97 to VOTER
I20240501 20:10:15.483721 9332 consensus_queue.cc:237] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:15.484547 7155 raft_consensus.cc:2454] T 5a7b2f01b5684df5800987591a390a17 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:10:15.485095 4299 leader_election.cc:302] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:15.485477 9143 raft_consensus.cc:2781] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election won for term 1
W20240501 20:10:15.491710 4301 leader_election.cc:341] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:10:15.492585 4301 leader_election.cc:341] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:15.494738 9521 raft_consensus.cc:348] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:15.495621 9522 tablet_bootstrap.cc:492] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:15.496985 8604 leader_election.cc:288] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:15.498407 9521 raft_consensus.cc:374] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:15.502136 9550 ts_tablet_manager.cc:902] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97: Initiating tablet copy from peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:15.498881 7022 raft_consensus.cc:2932] T a20287c2cab44a62b28037a463d4b49b P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:15.519634 4499 consensus_queue.cc:237] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:15.522542 7022 raft_consensus.cc:1261] T a20287c2cab44a62b28037a463d4b49b P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:15.523236 9143 raft_consensus.cc:686] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Becoming Leader. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Running, Role: LEADER
I20240501 20:10:15.523828 9319 consensus_queue.cc:1035] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:15.523836 9143 consensus_queue.cc:237] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:15.527452 9521 raft_consensus.cc:729] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:15.529781 4627 raft_consensus.cc:1261] T a20287c2cab44a62b28037a463d4b49b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:15.530704 7284 raft_consensus.cc:1261] T 23eaac7a50354947b6efa26ac5f01ae7 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:15.531363 9555 consensus_queue.cc:1035] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:15.531651 4365 raft_consensus.cc:1261] T 23eaac7a50354947b6efa26ac5f01ae7 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:15.533578 9319 consensus_queue.cc:1035] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:15.529270 9521 consensus_queue.cc:260] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:15.534922 9541 raft_consensus.cc:2932] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:15.535989 7021 raft_consensus.cc:2932] T a20287c2cab44a62b28037a463d4b49b P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:15.537866 9521 ts_tablet_manager.cc:1398] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.075s user 0.003s sys 0.000s
I20240501 20:10:15.539003 9522 tablet_bootstrap.cc:654] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:15.540057 9521 tablet_bootstrap.cc:492] T 7ecf2962444644cc8c5bdf5caf95c8c8 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:15.542443 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a20287c2cab44a62b28037a463d4b49b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:15.544991 4499 raft_consensus.cc:1261] T 23eaac7a50354947b6efa26ac5f01ae7 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:15.545763 4088 catalog_manager.cc:5484] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:15.548250 9558 consensus_queue.cc:1035] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:15.549082 9547 consensus_queue.cc:1035] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:15.567507 4346 tablet_service.cc:1514] Processing DeleteTablet for tablet a20287c2cab44a62b28037a463d4b49b with delete_type TABLET_DATA_TOMBSTONED (TS 10b47782558c41ee91f5e164a372049a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:15.568428 9521 tablet_bootstrap.cc:654] T 7ecf2962444644cc8c5bdf5caf95c8c8 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:15.569738 4088 catalog_manager.cc:5484] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a reported cstate change: term changed from 0 to 1, leader changed from <none> to 10b47782558c41ee91f5e164a372049a (127.29.219.66). New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:15.571401 4627 raft_consensus.cc:2932] T a20287c2cab44a62b28037a463d4b49b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:15.572430 9522 tablet_bootstrap.cc:492] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:10:15.572582 9522 ts_tablet_manager.cc:1367] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.077s user 0.003s sys 0.000s
I20240501 20:10:15.573262 9522 raft_consensus.cc:348] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:15.573462 9522 raft_consensus.cc:374] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:15.573520 9522 raft_consensus.cc:729] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:10:15.573725 9522 consensus_queue.cc:260] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:15.576751 9522 ts_tablet_manager.cc:1398] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:10:15.590219 4498 raft_consensus.cc:1261] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:10:15.591439 8851 consensus_queue.cc:1035] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:15.603163 9561 tablet_replica.cc:330] T a20287c2cab44a62b28037a463d4b49b P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:15.604012 9561 raft_consensus.cc:2229] T a20287c2cab44a62b28037a463d4b49b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:15.606506 9521 tablet_bootstrap.cc:492] T 7ecf2962444644cc8c5bdf5caf95c8c8 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:15.608283 9555 raft_consensus.cc:2932] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:15.609102 9521 ts_tablet_manager.cc:1367] T 7ecf2962444644cc8c5bdf5caf95c8c8 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.069s user 0.004s sys 0.000s
I20240501 20:10:15.609930 9521 raft_consensus.cc:348] T 7ecf2962444644cc8c5bdf5caf95c8c8 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:15.610139 9521 raft_consensus.cc:374] T 7ecf2962444644cc8c5bdf5caf95c8c8 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:15.610216 9521 raft_consensus.cc:729] T 7ecf2962444644cc8c5bdf5caf95c8c8 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:15.610819 9531 raft_consensus.cc:482] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:15.610929 9531 raft_consensus.cc:504] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:15.611584 9531 leader_election.cc:288] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:15.613422 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a7b2f01b5684df5800987591a390a17" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:15.613776 7155 raft_consensus.cc:2379] T 5a7b2f01b5684df5800987591a390a17 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ef0a4ada548446b3a531cb39a3c4e38d in current term 1: Already voted for candidate 10b47782558c41ee91f5e164a372049a in this term.
I20240501 20:10:15.610474 9521 consensus_queue.cc:260] T 7ecf2962444644cc8c5bdf5caf95c8c8 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:15.615025 9521 ts_tablet_manager.cc:1398] T 7ecf2962444644cc8c5bdf5caf95c8c8 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:10:15.615404 9521 tablet_bootstrap.cc:492] T ae106dc4a9d646cf8c6947e6895e03cd P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:15.617779 9521 tablet_bootstrap.cc:654] T ae106dc4a9d646cf8c6947e6895e03cd P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:15.619796 7285 raft_consensus.cc:2932] T 23eaac7a50354947b6efa26ac5f01ae7 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:15.614593 4367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a7b2f01b5684df5800987591a390a17" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:10:15.620371 9552 raft_consensus.cc:1050] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a: attempting to promote NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 to VOTER
I20240501 20:10:15.620697 9561 raft_consensus.cc:2258] T a20287c2cab44a62b28037a463d4b49b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:15.621212 9552 consensus_queue.cc:237] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:15.624806 6957 leader_election.cc:302] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef0a4ada548446b3a531cb39a3c4e38d; no voters: 10b47782558c41ee91f5e164a372049a, db0742ee3860468590eefef6ae6074fd
I20240501 20:10:15.625339 9531 raft_consensus.cc:3037] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:15.626873 9521 tablet_bootstrap.cc:492] T ae106dc4a9d646cf8c6947e6895e03cd P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:15.627053 9521 ts_tablet_manager.cc:1367] T ae106dc4a9d646cf8c6947e6895e03cd P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.001s
I20240501 20:10:15.628032 9521 raft_consensus.cc:348] T ae106dc4a9d646cf8c6947e6895e03cd P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:15.628387 9521 raft_consensus.cc:374] T ae106dc4a9d646cf8c6947e6895e03cd P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:15.628460 9521 raft_consensus.cc:729] T ae106dc4a9d646cf8c6947e6895e03cd P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:15.628716 9521 consensus_queue.cc:260] T ae106dc4a9d646cf8c6947e6895e03cd P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:15.629395 9561 ts_tablet_manager.cc:1875] T a20287c2cab44a62b28037a463d4b49b P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:15.631286 9531 raft_consensus.cc:2726] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:15.631963 9521 ts_tablet_manager.cc:1398] T ae106dc4a9d646cf8c6947e6895e03cd P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 20:10:15.632943 4091 catalog_manager.cc:5484] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:15.650830 9550 tablet_copy_client.cc:308] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.67:34595
I20240501 20:10:15.655064 4630 raft_consensus.cc:1261] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:10:15.655709 4519 tablet_copy_service.cc:140] P 3e974c03d3c546f9ab6c91e762a5426b: Received BeginTabletCopySession request for tablet 65b0048212a745c08ab24ead346bf74e from peer bceeae1c79964acabdae8ff771bf0c97 ({username='slave'} at 127.29.219.71:60861)
I20240501 20:10:15.655951 4519 tablet_copy_service.cc:161] P 3e974c03d3c546f9ab6c91e762a5426b: Beginning new tablet copy session on tablet 65b0048212a745c08ab24ead346bf74e from peer bceeae1c79964acabdae8ff771bf0c97 at {username='slave'} at 127.29.219.71:60861: session id = bceeae1c79964acabdae8ff771bf0c97-65b0048212a745c08ab24ead346bf74e
I20240501 20:10:15.655994 8851 consensus_queue.cc:1035] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:15.659195 4519 tablet_copy_source_session.cc:205] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:15.659204 4498 raft_consensus.cc:2932] T 23eaac7a50354947b6efa26ac5f01ae7 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:15.661278 9550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b0048212a745c08ab24ead346bf74e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:15.661675 4234 raft_consensus.cc:1261] T 91a2bf5ddc03431b963ab5da831fd340 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:15.662907 7419 raft_consensus.cc:1261] T 91a2bf5ddc03431b963ab5da831fd340 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:15.663486 9143 consensus_queue.cc:1035] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:15.664139 4499 raft_consensus.cc:1261] T 91a2bf5ddc03431b963ab5da831fd340 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:15.664131 9143 consensus_queue.cc:1035] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:15.667024 9552 consensus_queue.cc:1035] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:15.673254 9561 ts_tablet_manager.cc:1888] T a20287c2cab44a62b28037a463d4b49b P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:15.676964 4233 raft_consensus.cc:1261] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530 [term 2 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:10:15.678346 9403 consensus_queue.cc:1035] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:15.679222 9561 log.cc:1196] T a20287c2cab44a62b28037a463d4b49b P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/a20287c2cab44a62b28037a463d4b49b
I20240501 20:10:15.708748 8785 raft_consensus.cc:1050] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530: attempting to promote NON_VOTER db0742ee3860468590eefef6ae6074fd to VOTER
I20240501 20:10:15.709757 8785 consensus_queue.cc:237] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:15.716635 9512 raft_consensus.cc:1050] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97: attempting to promote NON_VOTER 10b47782558c41ee91f5e164a372049a to VOTER
I20240501 20:10:15.717562 9512 consensus_queue.cc:237] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } }
I20240501 20:10:15.731724 9552 raft_consensus.cc:2932] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:15.732465 4235 raft_consensus.cc:2932] T 91a2bf5ddc03431b963ab5da831fd340 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:15.736909 7420 raft_consensus.cc:2932] T 91a2bf5ddc03431b963ab5da831fd340 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:15.739815 4087 catalog_manager.cc:5484] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:15.743886 4630 consensus_queue.cc:237] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:15.748134 4367 consensus_queue.cc:237] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:15.749679 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet a20287c2cab44a62b28037a463d4b49b (table rebalance_test_table_17 [id=09dd73eba68c4f8499a786750d39c1cf]) successfully deleted
I20240501 20:10:15.746536 7285 raft_consensus.cc:1261] T 23eaac7a50354947b6efa26ac5f01ae7 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:15.750772 9555 consensus_queue.cc:1035] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:15.751458 4365 raft_consensus.cc:2932] T 23eaac7a50354947b6efa26ac5f01ae7 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:15.753436 9555 raft_consensus.cc:2932] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:15.758162 7285 raft_consensus.cc:2932] T 23eaac7a50354947b6efa26ac5f01ae7 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:15.765496 4235 raft_consensus.cc:1261] T 91a2bf5ddc03431b963ab5da831fd340 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:15.767746 9531 raft_consensus.cc:482] T 7ecf2962444644cc8c5bdf5caf95c8c8 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:15.767959 9531 raft_consensus.cc:504] T 7ecf2962444644cc8c5bdf5caf95c8c8 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:15.768358 9143 consensus_queue.cc:1035] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:15.768689 9531 leader_election.cc:288] T 7ecf2962444644cc8c5bdf5caf95c8c8 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:15.768810 4087 catalog_manager.cc:5484] T 23eaac7a50354947b6efa26ac5f01ae7 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:15.769500 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ecf2962444644cc8c5bdf5caf95c8c8" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:15.769716 7284 raft_consensus.cc:2454] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:15.772137 7153 raft_consensus.cc:1261] T a8eefa22b6e74b92b9fef036dea46e3b P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:15.772449 7421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ecf2962444644cc8c5bdf5caf95c8c8" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:15.772702 7421 raft_consensus.cc:2454] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:15.773162 6956 leader_election.cc:302] T 7ecf2962444644cc8c5bdf5caf95c8c8 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:15.773470 4079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 23eaac7a50354947b6efa26ac5f01ae7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:15.773530 8790 consensus_queue.cc:1035] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:15.774200 9531 raft_consensus.cc:2781] T 7ecf2962444644cc8c5bdf5caf95c8c8 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:15.774307 9531 raft_consensus.cc:482] T 7ecf2962444644cc8c5bdf5caf95c8c8 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:15.774415 9531 raft_consensus.cc:3037] T 7ecf2962444644cc8c5bdf5caf95c8c8 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:15.775806 4499 raft_consensus.cc:1261] T a8eefa22b6e74b92b9fef036dea46e3b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:15.776521 4630 raft_consensus.cc:1261] T a8eefa22b6e74b92b9fef036dea46e3b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:15.777112 8790 consensus_queue.cc:1035] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:15.777223 9531 raft_consensus.cc:504] T 7ecf2962444644cc8c5bdf5caf95c8c8 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:15.777870 9550 tablet_copy_client.cc:783] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:15.778748 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet 23eaac7a50354947b6efa26ac5f01ae7 with delete_type TABLET_DATA_TOMBSTONED (TS 3e974c03d3c546f9ab6c91e762a5426b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:15.779325 9531 leader_election.cc:288] T 7ecf2962444644cc8c5bdf5caf95c8c8 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Requested vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:15.779403 7421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ecf2962444644cc8c5bdf5caf95c8c8" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:15.779613 7421 raft_consensus.cc:3037] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:15.781605 9143 raft_consensus.cc:2932] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:15.782248 7421 raft_consensus.cc:2454] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:15.782869 6958 leader_election.cc:302] T 7ecf2962444644cc8c5bdf5caf95c8c8 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:15.783246 9531 raft_consensus.cc:2781] T 7ecf2962444644cc8c5bdf5caf95c8c8 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:15.783418 9531 raft_consensus.cc:686] T 7ecf2962444644cc8c5bdf5caf95c8c8 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Becoming Leader. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Running, Role: LEADER
I20240501 20:10:15.783807 9531 consensus_queue.cc:237] T 7ecf2962444644cc8c5bdf5caf95c8c8 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:15.786736 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 91a2bf5ddc03431b963ab5da831fd340 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:15.787137 8790 consensus_queue.cc:1035] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:15.787364 4235 raft_consensus.cc:2932] T 91a2bf5ddc03431b963ab5da831fd340 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:15.782598 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ecf2962444644cc8c5bdf5caf95c8c8" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:15.794706 7285 raft_consensus.cc:3037] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:15.795303 4087 catalog_manager.cc:5484] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:15.800077 7285 raft_consensus.cc:2454] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:15.800192 9158 raft_consensus.cc:482] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:15.800388 9158 raft_consensus.cc:504] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:15.801096 9158 leader_election.cc:288] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:15.801443 4477 tablet_service.cc:1514] Processing DeleteTablet for tablet 91a2bf5ddc03431b963ab5da831fd340 with delete_type TABLET_DATA_TOMBSTONED (TS 3e974c03d3c546f9ab6c91e762a5426b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:15.801584 4363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79af181b5f014a40a4c123b47a720dec" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:10:15.778236 9550 tablet_copy_client.cc:647] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:15.801901 4363 raft_consensus.cc:2454] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:15.803064 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79af181b5f014a40a4c123b47a720dec" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:15.803355 7155 raft_consensus.cc:2454] T 79af181b5f014a40a4c123b47a720dec P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:15.804031 6955 leader_election.cc:302] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db0742ee3860468590eefef6ae6074fd, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:15.804406 9158 raft_consensus.cc:2781] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:15.804507 9158 raft_consensus.cc:482] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:15.804633 9158 raft_consensus.cc:3037] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:15.809562 9588 tablet_replica.cc:330] T 23eaac7a50354947b6efa26ac5f01ae7 P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:15.811821 9550 tablet_copy_client.cc:523] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:15.812038 9201 raft_consensus.cc:482] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:15.812201 9201 raft_consensus.cc:504] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:15.812958 9158 raft_consensus.cc:504] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:15.813144 9201 leader_election.cc:288] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:15.814339 7021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae106dc4a9d646cf8c6947e6895e03cd" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:15.814565 7021 raft_consensus.cc:2454] T ae106dc4a9d646cf8c6947e6895e03cd P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:15.815250 9588 raft_consensus.cc:2229] T 23eaac7a50354947b6efa26ac5f01ae7 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:15.815578 7090 leader_election.cc:302] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db0742ee3860468590eefef6ae6074fd, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:15.816123 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79af181b5f014a40a4c123b47a720dec" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:15.816375 7155 raft_consensus.cc:3037] T 79af181b5f014a40a4c123b47a720dec P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:15.817059 9201 raft_consensus.cc:2781] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:15.817175 9201 raft_consensus.cc:482] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:15.817258 9201 raft_consensus.cc:3037] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:15.817926 9588 raft_consensus.cc:2258] T 23eaac7a50354947b6efa26ac5f01ae7 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:15.819962 4364 raft_consensus.cc:1261] T f82a24aabb2b4386b5916653a97e548c P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:15.819929 9201 raft_consensus.cc:504] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:15.819958 9158 leader_election.cc:288] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Requested vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:15.820856 7419 raft_consensus.cc:1261] T 91a2bf5ddc03431b963ab5da831fd340 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:15.821059 7021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae106dc4a9d646cf8c6947e6895e03cd" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:15.821270 7021 raft_consensus.cc:3037] T ae106dc4a9d646cf8c6947e6895e03cd P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:15.821730 4363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79af181b5f014a40a4c123b47a720dec" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:10:15.821595 8604 consensus_queue.cc:1035] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:15.821928 4363 raft_consensus.cc:3037] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:15.823314 9201 leader_election.cc:288] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:15.823391 4232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae106dc4a9d646cf8c6947e6895e03cd" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:10:15.823566 4232 raft_consensus.cc:3037] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:15.823539 4235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae106dc4a9d646cf8c6947e6895e03cd" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:15.823431 7419 raft_consensus.cc:2932] T 91a2bf5ddc03431b963ab5da831fd340 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:15.824774 4363 raft_consensus.cc:2454] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:15.826201 4232 raft_consensus.cc:2454] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:15.827411 7021 raft_consensus.cc:2454] T ae106dc4a9d646cf8c6947e6895e03cd P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:15.827440 7090 leader_election.cc:302] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db0742ee3860468590eefef6ae6074fd, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:15.827749 9201 raft_consensus.cc:2781] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:15.828676 9201 raft_consensus.cc:686] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Becoming Leader. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Running, Role: LEADER
I20240501 20:10:15.829033 9201 consensus_queue.cc:237] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:15.829166 9514 consensus_queue.cc:1035] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:15.830890 6957 leader_election.cc:302] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:15.831485 9533 raft_consensus.cc:2781] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:15.831642 9533 raft_consensus.cc:686] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Becoming Leader. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Running, Role: LEADER
I20240501 20:10:15.831955 9533 consensus_queue.cc:237] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:15.833287 7155 raft_consensus.cc:2454] T 79af181b5f014a40a4c123b47a720dec P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:15.837566 4091 catalog_manager.cc:5484] T 7ecf2962444644cc8c5bdf5caf95c8c8 P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69). New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:15.838321 9588 ts_tablet_manager.cc:1875] T 23eaac7a50354947b6efa26ac5f01ae7 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:15.841454 7021 raft_consensus.cc:1261] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:15.842150 9514 consensus_queue.cc:1035] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:15.843248 7421 raft_consensus.cc:1261] T f82a24aabb2b4386b5916653a97e548c P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:15.843950 9514 consensus_queue.cc:1035] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:15.846963 9514 raft_consensus.cc:2932] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:15.852234 4363 raft_consensus.cc:2932] T f82a24aabb2b4386b5916653a97e548c P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:15.860380 8785 raft_consensus.cc:2932] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:15.866441 9550 tablet_bootstrap.cc:492] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:15.866184 4091 catalog_manager.cc:5484] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd reported cstate change: term changed from 0 to 1, leader changed from <none> to db0742ee3860468590eefef6ae6074fd (127.29.219.70). New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:15.867269 4088 catalog_manager.cc:5484] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69). New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:15.871495 7421 raft_consensus.cc:2932] T f82a24aabb2b4386b5916653a97e548c P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:15.880008 4091 catalog_manager.cc:5484] T f82a24aabb2b4386b5916653a97e548c P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:15.884215 4627 raft_consensus.cc:2932] T a8eefa22b6e74b92b9fef036dea46e3b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:15.887981 7285 consensus_queue.cc:237] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } }
I20240501 20:10:15.891726 4091 catalog_manager.cc:5484] T a8eefa22b6e74b92b9fef036dea46e3b P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:15.895570 4363 raft_consensus.cc:1261] T f82a24aabb2b4386b5916653a97e548c P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:15.898234 9598 consensus_queue.cc:1035] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:15.899559 4367 raft_consensus.cc:1261] T 23eaac7a50354947b6efa26ac5f01ae7 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:15.900220 7153 raft_consensus.cc:2932] T a8eefa22b6e74b92b9fef036dea46e3b P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:15.901247 9555 consensus_queue.cc:1035] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:15.902977 4367 raft_consensus.cc:2932] T 23eaac7a50354947b6efa26ac5f01ae7 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:15.906210 7421 raft_consensus.cc:1261] T f82a24aabb2b4386b5916653a97e548c P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:15.906930 9598 consensus_queue.cc:1035] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:15.913045 9512 raft_consensus.cc:2932] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:15.918411 4234 consensus_queue.cc:237] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:15.920830 4077 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f82a24aabb2b4386b5916653a97e548c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:15.921841 9143 consensus_queue.cc:1035] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:15.926896 7152 raft_consensus.cc:1261] T a8eefa22b6e74b92b9fef036dea46e3b P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:15.926647 4367 raft_consensus.cc:2932] T f82a24aabb2b4386b5916653a97e548c P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:15.927999 9600 consensus_queue.cc:1035] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:15.932032 4091 catalog_manager.cc:5484] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:15.932511 4627 raft_consensus.cc:1261] T a8eefa22b6e74b92b9fef036dea46e3b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:15.933521 8785 consensus_queue.cc:1035] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:15.933354 7421 raft_consensus.cc:2932] T f82a24aabb2b4386b5916653a97e548c P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:15.936651 8790 raft_consensus.cc:2932] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:15.941354 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a8eefa22b6e74b92b9fef036dea46e3b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:15.943812 7152 raft_consensus.cc:2932] T a8eefa22b6e74b92b9fef036dea46e3b P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:15.945292 4627 raft_consensus.cc:2932] T a8eefa22b6e74b92b9fef036dea46e3b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:15.950989 4087 catalog_manager.cc:5484] T a8eefa22b6e74b92b9fef036dea46e3b P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:10:15.952011 4169 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.69:46237, user_credentials={real_user=slave}} blocked reactor thread for 96043.7us
I20240501 20:10:15.953356 4499 raft_consensus.cc:2932] T a8eefa22b6e74b92b9fef036dea46e3b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:15.955588 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet a8eefa22b6e74b92b9fef036dea46e3b with delete_type TABLET_DATA_TOMBSTONED (TS 3e974c03d3c546f9ab6c91e762a5426b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:15.957530 7002 tablet_service.cc:1514] Processing DeleteTablet for tablet f82a24aabb2b4386b5916653a97e548c with delete_type TABLET_DATA_TOMBSTONED (TS ef0a4ada548446b3a531cb39a3c4e38d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:15.962904 9588 ts_tablet_manager.cc:1888] T 23eaac7a50354947b6efa26ac5f01ae7 P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:15.963102 9588 log.cc:1196] T 23eaac7a50354947b6efa26ac5f01ae7 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/23eaac7a50354947b6efa26ac5f01ae7
I20240501 20:10:15.964040 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet 23eaac7a50354947b6efa26ac5f01ae7 (table rebalance_test_table_3 [id=5fb9ba92237649f89c99e4487cd45f29]) successfully deleted
I20240501 20:10:15.977914 9588 tablet_replica.cc:330] T 91a2bf5ddc03431b963ab5da831fd340 P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:15.980566 9588 raft_consensus.cc:2229] T 91a2bf5ddc03431b963ab5da831fd340 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:15.980921 9588 pending_rounds.cc:62] T 91a2bf5ddc03431b963ab5da831fd340 P 3e974c03d3c546f9ab6c91e762a5426b: Trying to abort 1 pending ops.
I20240501 20:10:15.981076 9588 pending_rounds.cc:68] T 91a2bf5ddc03431b963ab5da831fd340 P 3e974c03d3c546f9ab6c91e762a5426b: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022977907182333952 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "91a2bf5ddc03431b963ab5da831fd340" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } } }
I20240501 20:10:15.983183 8604 consensus_queue.cc:1035] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:15.988384 9604 tablet_replica.cc:330] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d: stopping tablet replica
I20240501 20:10:15.988759 9604 raft_consensus.cc:2229] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:15.989055 9604 pending_rounds.cc:62] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d: Trying to abort 1 pending ops.
I20240501 20:10:15.989236 9604 pending_rounds.cc:68] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7022977907576561664 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "f82a24aabb2b4386b5916653a97e548c" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } } }
I20240501 20:10:15.990234 9604 raft_consensus.cc:2894] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240501 20:10:15.990415 9604 raft_consensus.cc:2866] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:10:15.990463 9588 raft_consensus.cc:2894] T 91a2bf5ddc03431b963ab5da831fd340 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 20:10:15.990664 9604 raft_consensus.cc:2258] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:15.990684 9588 raft_consensus.cc:2866] T 91a2bf5ddc03431b963ab5da831fd340 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:10:15.990799 9588 raft_consensus.cc:2258] T 91a2bf5ddc03431b963ab5da831fd340 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:15.992015 9588 ts_tablet_manager.cc:1875] T 91a2bf5ddc03431b963ab5da831fd340 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:15.992945 9604 ts_tablet_manager.cc:1875] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:16.003408 9604 ts_tablet_manager.cc:1888] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:16.003717 9604 log.cc:1196] T f82a24aabb2b4386b5916653a97e548c P ef0a4ada548446b3a531cb39a3c4e38d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/wals/f82a24aabb2b4386b5916653a97e548c
I20240501 20:10:16.004601 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet f82a24aabb2b4386b5916653a97e548c (table rebalancer_extra_table_6 [id=f56b02450c6f407b827d0131f37c9ffa]) successfully deleted
I20240501 20:10:16.018595 9588 ts_tablet_manager.cc:1888] T 91a2bf5ddc03431b963ab5da831fd340 P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:16.026212 9588 log.cc:1196] T 91a2bf5ddc03431b963ab5da831fd340 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/91a2bf5ddc03431b963ab5da831fd340
I20240501 20:10:16.027175 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet 91a2bf5ddc03431b963ab5da831fd340 (table rebalance_test_table_13 [id=e21b0047cbf842c6833a78db81a34171]) successfully deleted
I20240501 20:10:16.027630 9588 tablet_replica.cc:330] T a8eefa22b6e74b92b9fef036dea46e3b P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:16.027854 9588 raft_consensus.cc:2229] T a8eefa22b6e74b92b9fef036dea46e3b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:16.028108 9588 raft_consensus.cc:2258] T a8eefa22b6e74b92b9fef036dea46e3b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:16.034415 9588 ts_tablet_manager.cc:1875] T a8eefa22b6e74b92b9fef036dea46e3b P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:16.042938 9588 ts_tablet_manager.cc:1888] T a8eefa22b6e74b92b9fef036dea46e3b P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:16.043148 9588 log.cc:1196] T a8eefa22b6e74b92b9fef036dea46e3b P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/a8eefa22b6e74b92b9fef036dea46e3b
I20240501 20:10:16.044059 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet a8eefa22b6e74b92b9fef036dea46e3b (table rebalance_test_table_7 [id=59dd8401cbd048cea30daf86baac8cfa]) successfully deleted
I20240501 20:10:16.072126 4088 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:16.073362 9550 tablet_bootstrap.cc:492] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:16.073910 9550 tablet_bootstrap.cc:492] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97: Bootstrap complete.
I20240501 20:10:16.074206 9550 ts_tablet_manager.cc:1367] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.208s user 0.047s sys 0.000s
I20240501 20:10:16.075748 9550 raft_consensus.cc:348] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:16.076043 9550 raft_consensus.cc:729] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: LEARNER
I20240501 20:10:16.076282 9550 consensus_queue.cc:260] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:16.076870 9550 ts_tablet_manager.cc:1398] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:16.078042 4519 tablet_copy_service.cc:342] P 3e974c03d3c546f9ab6c91e762a5426b: Request end of tablet copy session bceeae1c79964acabdae8ff771bf0c97-65b0048212a745c08ab24ead346bf74e received from {username='slave'} at 127.29.219.71:60861
I20240501 20:10:16.078214 4519 tablet_copy_service.cc:433] P 3e974c03d3c546f9ab6c91e762a5426b: ending tablet copy session bceeae1c79964acabdae8ff771bf0c97-65b0048212a745c08ab24ead346bf74e on tablet 65b0048212a745c08ab24ead346bf74e with peer bceeae1c79964acabdae8ff771bf0c97
I20240501 20:10:16.079879 7021 raft_consensus.cc:1261] T ae106dc4a9d646cf8c6947e6895e03cd P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:16.080837 4234 raft_consensus.cc:1261] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:16.081758 9546 consensus_queue.cc:1035] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:16.083434 9546 consensus_queue.cc:1035] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:16.088716 4365 consensus_queue.cc:237] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:16.091779 4499 raft_consensus.cc:1261] T ddcb6f5090f143c2b672e5630230878f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:16.092216 4627 raft_consensus.cc:1261] T ddcb6f5090f143c2b672e5630230878f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:16.092598 9553 consensus_queue.cc:1035] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:10:16.099077 4302 consensus_peers.cc:485] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a -> Peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): Couldn't send request to peer b9957d4086bc4a76b18ed9265a5739d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ddcb6f5090f143c2b672e5630230878f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:16.099316 9143 consensus_queue.cc:1035] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:16.101899 4093 catalog_manager.cc:5576] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941) does not have the latest schema for tablet ae106dc4a9d646cf8c6947e6895e03cd (table rebalancer_dynamic_table [id=04857c94591e4e16a9a3f7f9117b25ae]). Expected version 1 got 0
I20240501 20:10:16.102878 4091 catalog_manager.cc:5576] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237) does not have the latest schema for tablet ae106dc4a9d646cf8c6947e6895e03cd (table rebalancer_dynamic_table [id=04857c94591e4e16a9a3f7f9117b25ae]). Expected version 1 got 0
I20240501 20:10:16.103251 9635 tablet.cc:1722] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:16.104446 9635 tablet.cc:1620] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530: MemRowSet was empty: no flush needed.
I20240501 20:10:16.110443 4091 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=04857c94591e4e16a9a3f7f9117b25ae] alter complete (version 1)
I20240501 20:10:16.132439 8604 raft_consensus.cc:2932] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:16.133471 4631 raft_consensus.cc:2932] T ddcb6f5090f143c2b672e5630230878f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:16.140954 9638 tablet.cc:1722] T ae106dc4a9d646cf8c6947e6895e03cd P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:16.141508 9638 tablet.cc:1620] T ae106dc4a9d646cf8c6947e6895e03cd P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:16.142673 4091 catalog_manager.cc:5484] T ddcb6f5090f143c2b672e5630230878f P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:16.144425 4088 catalog_manager.cc:5576] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237) does not have the latest schema for tablet ae106dc4a9d646cf8c6947e6895e03cd (table rebalancer_dynamic_table [id=04857c94591e4e16a9a3f7f9117b25ae]). Expected version 1 got 0
I20240501 20:10:16.158111 4499 raft_consensus.cc:2932] T ddcb6f5090f143c2b672e5630230878f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:16.168560 9634 tablet.cc:1722] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:16.171209 9634 tablet.cc:1620] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:10:16.207574 9531 consensus_queue.cc:1035] T 7ecf2962444644cc8c5bdf5caf95c8c8 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:16.216722 4093 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:16.225133 9158 consensus_queue.cc:1035] T 7ecf2962444644cc8c5bdf5caf95c8c8 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:16.229704 9634 tablet.cc:1722] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:16.230289 9634 tablet.cc:1620] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:10:16.231961 9635 tablet.cc:1722] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:16.232551 9635 tablet.cc:1620] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530: MemRowSet was empty: no flush needed.
I20240501 20:10:16.233901 9638 tablet.cc:1722] T ae106dc4a9d646cf8c6947e6895e03cd P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:16.234370 9638 tablet.cc:1620] T ae106dc4a9d646cf8c6947e6895e03cd P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:16.239917 4088 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=04857c94591e4e16a9a3f7f9117b25ae] alter complete (version 2)
I20240501 20:10:16.242465 9158 consensus_queue.cc:1035] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:16.250968 4367 consensus_queue.cc:237] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:16.255601 9531 consensus_queue.cc:1035] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:16.262157 4499 raft_consensus.cc:1261] T 0bf6bcf89d444b339612d731af604d0a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:16.262504 4233 raft_consensus.cc:1261] T 0bf6bcf89d444b339612d731af604d0a P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:16.262991 9553 consensus_queue.cc:1035] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:16.263835 9143 consensus_queue.cc:1035] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:10:16.264156 4300 consensus_peers.cc:485] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a -> Peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Couldn't send request to peer bceeae1c79964acabdae8ff771bf0c97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0bf6bcf89d444b339612d731af604d0a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:16.276077 9553 raft_consensus.cc:2932] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:16.276859 4499 raft_consensus.cc:2932] T 0bf6bcf89d444b339612d731af604d0a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:16.277050 4234 raft_consensus.cc:2932] T 0bf6bcf89d444b339612d731af604d0a P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:16.285343 4093 catalog_manager.cc:5484] T 0bf6bcf89d444b339612d731af604d0a P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index -1 to 4, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:16.320178 7420 consensus_queue.cc:237] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:16.325158 4627 raft_consensus.cc:1261] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:10:16.329648 4233 raft_consensus.cc:1261] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530 [term 2 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:10:16.329797 9658 consensus_queue.cc:1035] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:16.331192 8851 consensus_queue.cc:1035] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:16.332996 9660 raft_consensus.cc:2932] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:16.336184 4233 raft_consensus.cc:2932] T 4f99f5d9d4bf44c39188c8b5d5362b98 P dca57823003e4729b769ca12367d7530 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:16.337844 4079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4f99f5d9d4bf44c39188c8b5d5362b98 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:16.340938 4091 catalog_manager.cc:5484] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: config changed from index 5 to 7, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New cstate: current_term: 2 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:16.343814 4627 raft_consensus.cc:2932] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:16.352990 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f99f5d9d4bf44c39188c8b5d5362b98 with delete_type TABLET_DATA_TOMBSTONED (TS 3e974c03d3c546f9ab6c91e762a5426b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:16.353446 9588 tablet_replica.cc:330] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:16.353471 4091 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:16.353683 9588 raft_consensus.cc:2229] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:16.353904 9588 raft_consensus.cc:2258] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:16.357142 9588 ts_tablet_manager.cc:1875] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:16.375321 9634 tablet.cc:1722] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:16.376992 9635 tablet.cc:1722] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:16.378261 9588 ts_tablet_manager.cc:1888] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:10:16.378475 9588 log.cc:1196] T 4f99f5d9d4bf44c39188c8b5d5362b98 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/4f99f5d9d4bf44c39188c8b5d5362b98
I20240501 20:10:16.379444 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet 4f99f5d9d4bf44c39188c8b5d5362b98 (table rebalance_test_table_19 [id=8de1e355c457486da8bba4c628d52c17]) successfully deleted
I20240501 20:10:16.387913 9638 tablet.cc:1722] T ae106dc4a9d646cf8c6947e6895e03cd P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:16.390663 4093 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table_new_name [id=04857c94591e4e16a9a3f7f9117b25ae] alter complete (version 3)
I20240501 20:10:16.407549 4093 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:16.418817 7267 tablet_service.cc:1467] Processing CreateTablet for tablet dfb7abcd009341acafded230f877c9f6 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=338f819ddccb42a3ae45044c701a6eb5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:16.419410 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb7abcd009341acafded230f877c9f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:16.419661 7002 tablet_service.cc:1467] Processing CreateTablet for tablet dfb7abcd009341acafded230f877c9f6 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=338f819ddccb42a3ae45044c701a6eb5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:16.420190 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb7abcd009341acafded230f877c9f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:16.421005 4609 tablet_service.cc:1467] Processing CreateTablet for tablet c5b4fbeda51f40b8ac54baf5fb241710 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=338f819ddccb42a3ae45044c701a6eb5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:16.421614 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b4fbeda51f40b8ac54baf5fb241710. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:16.424742 7135 tablet_service.cc:1467] Processing CreateTablet for tablet dfb7abcd009341acafded230f877c9f6 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=338f819ddccb42a3ae45044c701a6eb5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:16.425374 7135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb7abcd009341acafded230f877c9f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:16.437124 4478 tablet_service.cc:1467] Processing CreateTablet for tablet c5b4fbeda51f40b8ac54baf5fb241710 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=338f819ddccb42a3ae45044c701a6eb5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:16.437630 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b4fbeda51f40b8ac54baf5fb241710. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:16.438421 7401 tablet_service.cc:1467] Processing CreateTablet for tablet c5b4fbeda51f40b8ac54baf5fb241710 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=338f819ddccb42a3ae45044c701a6eb5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:16.439046 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b4fbeda51f40b8ac54baf5fb241710. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:16.439632 9677 tablet_bootstrap.cc:492] T dfb7abcd009341acafded230f877c9f6 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:16.440522 7400 tablet_service.cc:1467] Processing CreateTablet for tablet 743d7fd2ae014352a5681c2a8ee4447d (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=338f819ddccb42a3ae45044c701a6eb5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:16.441015 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743d7fd2ae014352a5681c2a8ee4447d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:16.441555 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 743d7fd2ae014352a5681c2a8ee4447d (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=338f819ddccb42a3ae45044c701a6eb5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:16.442102 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743d7fd2ae014352a5681c2a8ee4447d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:16.442384 9677 tablet_bootstrap.cc:654] T dfb7abcd009341acafded230f877c9f6 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:16.448664 7002 tablet_service.cc:1467] Processing CreateTablet for tablet 743d7fd2ae014352a5681c2a8ee4447d (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=338f819ddccb42a3ae45044c701a6eb5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:16.449404 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743d7fd2ae014352a5681c2a8ee4447d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:16.472350 7285 raft_consensus.cc:1203] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:16.485888 9681 tablet_bootstrap.cc:492] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:16.491639 9681 tablet_bootstrap.cc:654] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:16.493741 9682 tablet_bootstrap.cc:492] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:16.495162 9684 tablet_bootstrap.cc:492] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:16.495656 4093 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:16.496013 4093 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:16.497041 9678 tablet_bootstrap.cc:492] T dfb7abcd009341acafded230f877c9f6 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:16.497624 9680 tablet_bootstrap.cc:492] T c5b4fbeda51f40b8ac54baf5fb241710 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:16.499487 9677 tablet_bootstrap.cc:492] T dfb7abcd009341acafded230f877c9f6 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:16.499776 9677 ts_tablet_manager.cc:1367] T dfb7abcd009341acafded230f877c9f6 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.060s user 0.013s sys 0.023s
I20240501 20:10:16.500686 9678 tablet_bootstrap.cc:654] T dfb7abcd009341acafded230f877c9f6 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:16.500705 9682 tablet_bootstrap.cc:654] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:16.500825 9677 raft_consensus.cc:348] T dfb7abcd009341acafded230f877c9f6 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:16.501251 9677 raft_consensus.cc:374] T dfb7abcd009341acafded230f877c9f6 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:16.501292 4093 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet ae106dc4a9d646cf8c6947e6895e03cd
I20240501 20:10:16.501581 9677 raft_consensus.cc:729] T dfb7abcd009341acafded230f877c9f6 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:16.502162 9677 consensus_queue.cc:260] T dfb7abcd009341acafded230f877c9f6 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:16.503089 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet ae106dc4a9d646cf8c6947e6895e03cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:16 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:16.503527 9684 tablet_bootstrap.cc:654] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:16.503736 7002 tablet_service.cc:1514] Processing DeleteTablet for tablet ae106dc4a9d646cf8c6947e6895e03cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:16 UTC) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:16.504098 9604 tablet_replica.cc:330] T ae106dc4a9d646cf8c6947e6895e03cd P ef0a4ada548446b3a531cb39a3c4e38d: stopping tablet replica
I20240501 20:10:16.504338 9604 raft_consensus.cc:2229] T ae106dc4a9d646cf8c6947e6895e03cd P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:16.504351 7135 tablet_service.cc:1514] Processing DeleteTablet for tablet ae106dc4a9d646cf8c6947e6895e03cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:16 UTC) from {username='slave'} at 127.0.0.1:52036
I20240501 20:10:16.504588 9604 raft_consensus.cc:2258] T ae106dc4a9d646cf8c6947e6895e03cd P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:16.505314 9677 ts_tablet_manager.cc:1398] T dfb7abcd009341acafded230f877c9f6 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240501 20:10:16.505759 9604 ts_tablet_manager.cc:1875] T ae106dc4a9d646cf8c6947e6895e03cd P ef0a4ada548446b3a531cb39a3c4e38d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:16.512727 9682 tablet_bootstrap.cc:492] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:16.514433 9682 ts_tablet_manager.cc:1367] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240501 20:10:16.515998 9682 raft_consensus.cc:348] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:16.517935 9604 ts_tablet_manager.cc:1888] T ae106dc4a9d646cf8c6947e6895e03cd P ef0a4ada548446b3a531cb39a3c4e38d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:16.518103 9604 log.cc:1196] T ae106dc4a9d646cf8c6947e6895e03cd P ef0a4ada548446b3a531cb39a3c4e38d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/wals/ae106dc4a9d646cf8c6947e6895e03cd
I20240501 20:10:16.518688 9604 ts_tablet_manager.cc:1909] T ae106dc4a9d646cf8c6947e6895e03cd P ef0a4ada548446b3a531cb39a3c4e38d: Deleting consensus metadata
I20240501 20:10:16.518716 9682 raft_consensus.cc:374] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:16.519217 9682 raft_consensus.cc:729] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:16.519232 9680 tablet_bootstrap.cc:654] T c5b4fbeda51f40b8ac54baf5fb241710 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:16.523005 9682 consensus_queue.cc:260] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:16.524593 9677 tablet_bootstrap.cc:492] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:16.526399 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet ae106dc4a9d646cf8c6947e6895e03cd (table rebalancer_dynamic_table_new_name [id=04857c94591e4e16a9a3f7f9117b25ae]) successfully deleted
I20240501 20:10:16.526556 9687 tablet_replica.cc:330] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:16.527031 9688 tablet_replica.cc:330] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd: stopping tablet replica
I20240501 20:10:16.527302 9687 raft_consensus.cc:2229] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:16.527524 9687 raft_consensus.cc:2258] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:16.528438 9681 tablet_bootstrap.cc:492] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:16.528693 9681 ts_tablet_manager.cc:1367] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.000s
I20240501 20:10:16.529690 9681 raft_consensus.cc:348] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:16.530097 9681 raft_consensus.cc:374] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:16.530257 9681 raft_consensus.cc:729] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:16.530576 9681 consensus_queue.cc:260] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:16.531744 9688 raft_consensus.cc:2229] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:16.532200 9688 raft_consensus.cc:2258] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:16.532801 9678 tablet_bootstrap.cc:492] T dfb7abcd009341acafded230f877c9f6 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:16.533042 9677 tablet_bootstrap.cc:654] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:16.533018 9678 ts_tablet_manager.cc:1367] T dfb7abcd009341acafded230f877c9f6 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.000s
I20240501 20:10:16.533687 9682 ts_tablet_manager.cc:1398] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240501 20:10:16.534417 9678 raft_consensus.cc:348] T dfb7abcd009341acafded230f877c9f6 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:16.534744 9687 ts_tablet_manager.cc:1875] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:16.534726 9678 raft_consensus.cc:374] T dfb7abcd009341acafded230f877c9f6 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:16.534970 9678 raft_consensus.cc:729] T dfb7abcd009341acafded230f877c9f6 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:16.535287 9678 consensus_queue.cc:260] T dfb7abcd009341acafded230f877c9f6 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:16.536595 9688 ts_tablet_manager.cc:1875] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:16.537772 9681 ts_tablet_manager.cc:1398] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240501 20:10:16.544270 9684 tablet_bootstrap.cc:492] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:16.544602 9684 ts_tablet_manager.cc:1367] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.004s
I20240501 20:10:16.545737 9684 raft_consensus.cc:348] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:16.546072 9684 raft_consensus.cc:374] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:16.546250 9684 raft_consensus.cc:729] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:16.546583 9684 consensus_queue.cc:260] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:16.547053 9678 ts_tablet_manager.cc:1398] T dfb7abcd009341acafded230f877c9f6 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240501 20:10:16.547927 9681 tablet_bootstrap.cc:492] T 743d7fd2ae014352a5681c2a8ee4447d P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:16.548795 9684 ts_tablet_manager.cc:1398] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:10:16.549173 9684 tablet_bootstrap.cc:492] T c5b4fbeda51f40b8ac54baf5fb241710 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:16.549368 9680 tablet_bootstrap.cc:492] T c5b4fbeda51f40b8ac54baf5fb241710 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:10:16.549556 9680 ts_tablet_manager.cc:1367] T c5b4fbeda51f40b8ac54baf5fb241710 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.052s user 0.000s sys 0.005s
I20240501 20:10:16.550664 9680 raft_consensus.cc:348] T c5b4fbeda51f40b8ac54baf5fb241710 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:16.550951 9680 raft_consensus.cc:374] T c5b4fbeda51f40b8ac54baf5fb241710 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:16.551072 9680 raft_consensus.cc:729] T c5b4fbeda51f40b8ac54baf5fb241710 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:10:16.551398 9201 raft_consensus.cc:482] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:16.551354 9680 consensus_queue.cc:260] T c5b4fbeda51f40b8ac54baf5fb241710 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:16.551611 9201 raft_consensus.cc:504] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:16.552044 9680 ts_tablet_manager.cc:1398] T c5b4fbeda51f40b8ac54baf5fb241710 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:10:16.553274 9681 tablet_bootstrap.cc:654] T 743d7fd2ae014352a5681c2a8ee4447d P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:16.555250 9201 leader_election.cc:288] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:16.555691 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb7abcd009341acafded230f877c9f6" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:16.555946 7284 raft_consensus.cc:2454] T dfb7abcd009341acafded230f877c9f6 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:16.556234 9684 tablet_bootstrap.cc:654] T c5b4fbeda51f40b8ac54baf5fb241710 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:16.556465 7089 leader_election.cc:302] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:16.557305 9201 raft_consensus.cc:2781] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:16.557428 9201 raft_consensus.cc:482] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:16.557579 9201 raft_consensus.cc:3037] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:16.557791 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb7abcd009341acafded230f877c9f6" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:16.558027 7022 raft_consensus.cc:2454] T dfb7abcd009341acafded230f877c9f6 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:16.560803 9687 ts_tablet_manager.cc:1888] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:16.561048 9687 log.cc:1196] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/ae106dc4a9d646cf8c6947e6895e03cd
I20240501 20:10:16.561559 9687 ts_tablet_manager.cc:1909] T ae106dc4a9d646cf8c6947e6895e03cd P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:16.562952 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet ae106dc4a9d646cf8c6947e6895e03cd (table rebalancer_dynamic_table_new_name [id=04857c94591e4e16a9a3f7f9117b25ae]) successfully deleted
I20240501 20:10:16.563338 9201 raft_consensus.cc:504] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:16.565395 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb7abcd009341acafded230f877c9f6" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:16.565725 7022 raft_consensus.cc:3037] T dfb7abcd009341acafded230f877c9f6 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:16.565806 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb7abcd009341acafded230f877c9f6" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:16.566045 7284 raft_consensus.cc:3037] T dfb7abcd009341acafded230f877c9f6 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:16.569273 7284 raft_consensus.cc:2454] T dfb7abcd009341acafded230f877c9f6 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:16.571352 7022 raft_consensus.cc:2454] T dfb7abcd009341acafded230f877c9f6 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:16.572537 9695 ts_tablet_manager.cc:902] T ddcb6f5090f143c2b672e5630230878f P b9957d4086bc4a76b18ed9265a5739d5: Initiating tablet copy from peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:16.573674 9681 tablet_bootstrap.cc:492] T 743d7fd2ae014352a5681c2a8ee4447d P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:16.573943 9681 ts_tablet_manager.cc:1367] T 743d7fd2ae014352a5681c2a8ee4447d P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.002s
I20240501 20:10:16.575333 9681 raft_consensus.cc:348] T 743d7fd2ae014352a5681c2a8ee4447d P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:16.575891 9681 raft_consensus.cc:374] T 743d7fd2ae014352a5681c2a8ee4447d P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:16.576958 9677 tablet_bootstrap.cc:492] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:16.577320 9677 ts_tablet_manager.cc:1367] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.006s
I20240501 20:10:16.579279 9677 raft_consensus.cc:348] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:16.579751 9677 raft_consensus.cc:374] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:16.579902 9677 raft_consensus.cc:729] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:16.580641 9677 consensus_queue.cc:260] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:16.582067 9684 tablet_bootstrap.cc:492] T c5b4fbeda51f40b8ac54baf5fb241710 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:16.582283 9684 ts_tablet_manager.cc:1367] T c5b4fbeda51f40b8ac54baf5fb241710 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.000s
I20240501 20:10:16.582950 9677 ts_tablet_manager.cc:1398] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:10:16.580817 9695 tablet_copy_client.cc:308] T ddcb6f5090f143c2b672e5630230878f P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.66:40689
I20240501 20:10:16.583317 9681 raft_consensus.cc:729] T 743d7fd2ae014352a5681c2a8ee4447d P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:16.583691 9681 consensus_queue.cc:260] T 743d7fd2ae014352a5681c2a8ee4447d P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:16.584182 4388 tablet_copy_service.cc:140] P 10b47782558c41ee91f5e164a372049a: Received BeginTabletCopySession request for tablet ddcb6f5090f143c2b672e5630230878f from peer b9957d4086bc4a76b18ed9265a5739d5 ({username='slave'} at 127.29.219.72:44421)
I20240501 20:10:16.584381 4388 tablet_copy_service.cc:161] P 10b47782558c41ee91f5e164a372049a: Beginning new tablet copy session on tablet ddcb6f5090f143c2b672e5630230878f from peer b9957d4086bc4a76b18ed9265a5739d5 at {username='slave'} at 127.29.219.72:44421: session id = b9957d4086bc4a76b18ed9265a5739d5-ddcb6f5090f143c2b672e5630230878f
I20240501 20:10:16.584419 9688 ts_tablet_manager.cc:1888] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:16.584612 9688 log.cc:1196] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/wals/ae106dc4a9d646cf8c6947e6895e03cd
I20240501 20:10:16.585143 9688 ts_tablet_manager.cc:1909] T ae106dc4a9d646cf8c6947e6895e03cd P db0742ee3860468590eefef6ae6074fd: Deleting consensus metadata
I20240501 20:10:16.585706 9681 ts_tablet_manager.cc:1398] T 743d7fd2ae014352a5681c2a8ee4447d P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20240501 20:10:16.586655 9684 raft_consensus.cc:348] T c5b4fbeda51f40b8ac54baf5fb241710 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:16.586923 9684 raft_consensus.cc:374] T c5b4fbeda51f40b8ac54baf5fb241710 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:16.587002 9684 raft_consensus.cc:729] T c5b4fbeda51f40b8ac54baf5fb241710 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:16.587296 9684 consensus_queue.cc:260] T c5b4fbeda51f40b8ac54baf5fb241710 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:16.587932 9684 ts_tablet_manager.cc:1398] T c5b4fbeda51f40b8ac54baf5fb241710 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:10:16.588603 7089 leader_election.cc:302] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:16.588925 9033 raft_consensus.cc:2781] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:16.589035 9201 leader_election.cc:288] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Requested vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:16.589107 9033 raft_consensus.cc:686] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Becoming Leader. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Running, Role: LEADER
I20240501 20:10:16.589386 9033 consensus_queue.cc:237] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:16.589553 4388 tablet_copy_source_session.cc:205] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:16.591156 4076 catalog_manager.cc:4830] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): tablet ae106dc4a9d646cf8c6947e6895e03cd (table rebalancer_dynamic_table_new_name [id=04857c94591e4e16a9a3f7f9117b25ae]) successfully deleted
I20240501 20:10:16.591348 9695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddcb6f5090f143c2b672e5630230878f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:16.600718 9695 tablet_copy_client.cc:783] T ddcb6f5090f143c2b672e5630230878f P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:16.601167 9695 tablet_copy_client.cc:647] T ddcb6f5090f143c2b672e5630230878f P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:16.605760 9695 tablet_copy_client.cc:523] T ddcb6f5090f143c2b672e5630230878f P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:16.606565 4093 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:16.611347 9695 tablet_bootstrap.cc:492] T ddcb6f5090f143c2b672e5630230878f P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:16.613749 4091 catalog_manager.cc:5484] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd reported cstate change: term changed from 0 to 1, leader changed from <none> to db0742ee3860468590eefef6ae6074fd (127.29.219.70). New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:16.622967 9660 raft_consensus.cc:482] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:16.623205 9660 raft_consensus.cc:504] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:16.623468 7002 tablet_service.cc:1467] Processing CreateTablet for tablet ca0fadee796b431f948266c96ab0c93d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1c0e2cb9c07f47a2b6505918ddbd35fc]), partition=
I20240501 20:10:16.623927 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca0fadee796b431f948266c96ab0c93d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:16.623957 9660 leader_election.cc:288] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:16.625165 7267 tablet_service.cc:1467] Processing CreateTablet for tablet ca0fadee796b431f948266c96ab0c93d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1c0e2cb9c07f47a2b6505918ddbd35fc]), partition=
I20240501 20:10:16.625698 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca0fadee796b431f948266c96ab0c93d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:16.627432 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b4fbeda51f40b8ac54baf5fb241710" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:16.627828 7135 tablet_service.cc:1467] Processing CreateTablet for tablet ca0fadee796b431f948266c96ab0c93d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1c0e2cb9c07f47a2b6505918ddbd35fc]), partition=
I20240501 20:10:16.627763 4499 raft_consensus.cc:2454] T c5b4fbeda51f40b8ac54baf5fb241710 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:10:16.628198 7135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca0fadee796b431f948266c96ab0c93d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:16.628548 7355 leader_election.cc:302] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, b9957d4086bc4a76b18ed9265a5739d5; no voters:
I20240501 20:10:16.628960 9660 raft_consensus.cc:2781] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:16.629086 9660 raft_consensus.cc:482] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:16.629227 9660 raft_consensus.cc:3037] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:16.632426 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b4fbeda51f40b8ac54baf5fb241710" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:16.632745 4627 raft_consensus.cc:2454] T c5b4fbeda51f40b8ac54baf5fb241710 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:10:16.640105 9682 tablet_bootstrap.cc:492] T ca0fadee796b431f948266c96ab0c93d P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:16.642017 9682 tablet_bootstrap.cc:654] T ca0fadee796b431f948266c96ab0c93d P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:16.644992 9660 raft_consensus.cc:504] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:16.645918 9677 tablet_bootstrap.cc:492] T ca0fadee796b431f948266c96ab0c93d P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:16.647802 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b4fbeda51f40b8ac54baf5fb241710" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:10:16.648038 9677 tablet_bootstrap.cc:654] T ca0fadee796b431f948266c96ab0c93d P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:16.648154 4627 raft_consensus.cc:3037] T c5b4fbeda51f40b8ac54baf5fb241710 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:16.652925 4627 raft_consensus.cc:2454] T c5b4fbeda51f40b8ac54baf5fb241710 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:16.653530 7356 leader_election.cc:302] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, b9957d4086bc4a76b18ed9265a5739d5; no voters:
I20240501 20:10:16.653934 9658 raft_consensus.cc:2781] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:16.654127 9658 raft_consensus.cc:686] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Becoming Leader. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Running, Role: LEADER
I20240501 20:10:16.654510 9658 consensus_queue.cc:237] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:16.658591 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b4fbeda51f40b8ac54baf5fb241710" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:10:16.658919 4499 raft_consensus.cc:3037] T c5b4fbeda51f40b8ac54baf5fb241710 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:16.662056 4499 raft_consensus.cc:2454] T c5b4fbeda51f40b8ac54baf5fb241710 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:16.666735 9660 leader_election.cc:288] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:16.669045 9678 tablet_bootstrap.cc:492] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:16.670735 9677 tablet_bootstrap.cc:492] T ca0fadee796b431f948266c96ab0c93d P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:16.670953 9677 ts_tablet_manager.cc:1367] T ca0fadee796b431f948266c96ab0c93d P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.007s
I20240501 20:10:16.671870 9678 tablet_bootstrap.cc:654] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:16.671936 9677 raft_consensus.cc:348] T ca0fadee796b431f948266c96ab0c93d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:16.672189 9677 raft_consensus.cc:374] T ca0fadee796b431f948266c96ab0c93d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:16.672297 9677 raft_consensus.cc:729] T ca0fadee796b431f948266c96ab0c93d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:16.673055 9677 consensus_queue.cc:260] T ca0fadee796b431f948266c96ab0c93d P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:16.680275 4088 catalog_manager.cc:5484] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72). New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:16.687770 9677 ts_tablet_manager.cc:1398] T ca0fadee796b431f948266c96ab0c93d P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240501 20:10:16.696775 9682 tablet_bootstrap.cc:492] T ca0fadee796b431f948266c96ab0c93d P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:16.697254 9682 ts_tablet_manager.cc:1367] T ca0fadee796b431f948266c96ab0c93d P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.010s
I20240501 20:10:16.698796 9682 raft_consensus.cc:348] T ca0fadee796b431f948266c96ab0c93d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:16.699272 9682 raft_consensus.cc:374] T ca0fadee796b431f948266c96ab0c93d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:16.699433 9682 raft_consensus.cc:729] T ca0fadee796b431f948266c96ab0c93d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:16.699882 9682 consensus_queue.cc:260] T ca0fadee796b431f948266c96ab0c93d P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:16.705221 9682 ts_tablet_manager.cc:1398] T ca0fadee796b431f948266c96ab0c93d P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 20:10:16.711781 9678 tablet_bootstrap.cc:492] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:16.712134 9678 ts_tablet_manager.cc:1367] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.010s
I20240501 20:10:16.713124 9678 raft_consensus.cc:348] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:16.713532 9678 raft_consensus.cc:374] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:16.713673 9678 raft_consensus.cc:729] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:16.714254 9678 consensus_queue.cc:260] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:16.715670 9678 ts_tablet_manager.cc:1398] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:10:16.715804 9158 raft_consensus.cc:482] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:16.716030 9158 raft_consensus.cc:504] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:16.717046 9158 leader_election.cc:288] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:16.720177 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743d7fd2ae014352a5681c2a8ee4447d" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:16.720428 4627 raft_consensus.cc:2454] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:16.721313 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743d7fd2ae014352a5681c2a8ee4447d" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:16.721549 7420 raft_consensus.cc:2454] T 743d7fd2ae014352a5681c2a8ee4447d P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:16.722005 6958 leader_election.cc:302] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:16.722481 9533 raft_consensus.cc:2781] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:16.722604 9533 raft_consensus.cc:482] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:16.722849 9533 raft_consensus.cc:3037] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:16.725944 9533 raft_consensus.cc:504] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:16.733003 9533 leader_election.cc:288] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:16.733860 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743d7fd2ae014352a5681c2a8ee4447d" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:10:16.734151 4627 raft_consensus.cc:3037] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:16.737387 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743d7fd2ae014352a5681c2a8ee4447d" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:16.737673 7420 raft_consensus.cc:3037] T 743d7fd2ae014352a5681c2a8ee4447d P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:16.742183 7420 raft_consensus.cc:2454] T 743d7fd2ae014352a5681c2a8ee4447d P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:16.745824 4627 raft_consensus.cc:2454] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:16.744732 6958 leader_election.cc:302] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:16.746694 9158 raft_consensus.cc:2781] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:16.746862 9158 raft_consensus.cc:686] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Becoming Leader. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Running, Role: LEADER
I20240501 20:10:16.751749 9158 consensus_queue.cc:237] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:16.763490 4091 catalog_manager.cc:5484] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69). New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:16.784705 9695 tablet_bootstrap.cc:492] T ddcb6f5090f143c2b672e5630230878f P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:16.785310 9695 tablet_bootstrap.cc:492] T ddcb6f5090f143c2b672e5630230878f P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap complete.
I20240501 20:10:16.785648 9695 ts_tablet_manager.cc:1367] T ddcb6f5090f143c2b672e5630230878f P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.174s user 0.058s sys 0.003s
I20240501 20:10:16.790143 9695 raft_consensus.cc:348] T ddcb6f5090f143c2b672e5630230878f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:16.790441 9695 raft_consensus.cc:729] T ddcb6f5090f143c2b672e5630230878f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: LEARNER
I20240501 20:10:16.791352 9695 consensus_queue.cc:260] T ddcb6f5090f143c2b672e5630230878f P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:16.793406 9695 ts_tablet_manager.cc:1398] T ddcb6f5090f143c2b672e5630230878f P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 20:10:16.794121 4388 tablet_copy_service.cc:342] P 10b47782558c41ee91f5e164a372049a: Request end of tablet copy session b9957d4086bc4a76b18ed9265a5739d5-ddcb6f5090f143c2b672e5630230878f received from {username='slave'} at 127.29.219.72:44421
I20240501 20:10:16.794282 4388 tablet_copy_service.cc:433] P 10b47782558c41ee91f5e164a372049a: ending tablet copy session b9957d4086bc4a76b18ed9265a5739d5-ddcb6f5090f143c2b672e5630230878f on tablet ddcb6f5090f143c2b672e5630230878f with peer b9957d4086bc4a76b18ed9265a5739d5
I20240501 20:10:16.799177 9692 raft_consensus.cc:482] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:16.799371 9692 raft_consensus.cc:504] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:16.800524 9692 leader_election.cc:288] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:16.800793 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0fadee796b431f948266c96ab0c93d" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:16.801067 7022 raft_consensus.cc:2454] T ca0fadee796b431f948266c96ab0c93d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:16.801923 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0fadee796b431f948266c96ab0c93d" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:16.802150 7153 raft_consensus.cc:2454] T ca0fadee796b431f948266c96ab0c93d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:16.802641 7222 leader_election.cc:302] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:16.805881 9598 raft_consensus.cc:2781] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:16.806003 9598 raft_consensus.cc:482] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:16.806133 9598 raft_consensus.cc:3037] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:16.809041 9598 raft_consensus.cc:504] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:16.812070 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0fadee796b431f948266c96ab0c93d" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:16.812358 7153 raft_consensus.cc:3037] T ca0fadee796b431f948266c96ab0c93d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:16.815693 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca0fadee796b431f948266c96ab0c93d" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:16.815932 7022 raft_consensus.cc:3037] T ca0fadee796b431f948266c96ab0c93d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:16.824440 7022 raft_consensus.cc:2454] T ca0fadee796b431f948266c96ab0c93d P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:16.825053 9598 leader_election.cc:288] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:16.825837 7222 leader_election.cc:302] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:16.827085 7153 raft_consensus.cc:2454] T ca0fadee796b431f948266c96ab0c93d P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:16.827526 9598 raft_consensus.cc:2781] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:16.827683 9598 raft_consensus.cc:686] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Becoming Leader. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Running, Role: LEADER
I20240501 20:10:16.827934 9598 consensus_queue.cc:237] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:16.848052 9719 ts_tablet_manager.cc:902] T 0bf6bcf89d444b339612d731af604d0a P bceeae1c79964acabdae8ff771bf0c97: Initiating tablet copy from peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:16.877282 4091 catalog_manager.cc:5484] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71). New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:16.883587 9719 tablet_copy_client.cc:308] T 0bf6bcf89d444b339612d731af604d0a P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.66:40689
I20240501 20:10:16.884328 4388 tablet_copy_service.cc:140] P 10b47782558c41ee91f5e164a372049a: Received BeginTabletCopySession request for tablet 0bf6bcf89d444b339612d731af604d0a from peer bceeae1c79964acabdae8ff771bf0c97 ({username='slave'} at 127.29.219.71:41437)
I20240501 20:10:16.884518 4388 tablet_copy_service.cc:161] P 10b47782558c41ee91f5e164a372049a: Beginning new tablet copy session on tablet 0bf6bcf89d444b339612d731af604d0a from peer bceeae1c79964acabdae8ff771bf0c97 at {username='slave'} at 127.29.219.71:41437: session id = bceeae1c79964acabdae8ff771bf0c97-0bf6bcf89d444b339612d731af604d0a
I20240501 20:10:16.887465 4388 tablet_copy_source_session.cc:205] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:16.890144 9719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf6bcf89d444b339612d731af604d0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:16.899363 9719 tablet_copy_client.cc:783] T 0bf6bcf89d444b339612d731af604d0a P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:16.899847 9719 tablet_copy_client.cc:647] T 0bf6bcf89d444b339612d731af604d0a P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:16.901892 9719 tablet_copy_client.cc:523] T 0bf6bcf89d444b339612d731af604d0a P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:16.905280 9719 tablet_bootstrap.cc:492] T 0bf6bcf89d444b339612d731af604d0a P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:16.910765 7155 consensus_queue.cc:237] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:16.915927 7420 raft_consensus.cc:1261] T 35c843bebb484f6cb655c146a02261ba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:16.916121 7022 raft_consensus.cc:1261] T 35c843bebb484f6cb655c146a02261ba P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:16.916654 9546 consensus_queue.cc:1035] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:16.917104 9033 consensus_queue.cc:1035] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:16.917454 7090 consensus_peers.cc:485] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35c843bebb484f6cb655c146a02261ba. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:16.920949 9319 raft_consensus.cc:1050] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b: attempting to promote NON_VOTER bceeae1c79964acabdae8ff771bf0c97 to VOTER
I20240501 20:10:16.922130 9319 consensus_queue.cc:237] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:16.934826 9033 raft_consensus.cc:2932] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:16.936309 4627 raft_consensus.cc:1261] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:16.936339 4367 raft_consensus.cc:1261] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:16.937208 9739 consensus_queue.cc:1035] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:16.937633 7285 raft_consensus.cc:1261] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:16.938316 9683 consensus_queue.cc:1035] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:16.938730 9169 consensus_queue.cc:1035] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:16.940297 7420 raft_consensus.cc:2932] T 35c843bebb484f6cb655c146a02261ba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:16.951191 4091 catalog_manager.cc:5484] T 35c843bebb484f6cb655c146a02261ba P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:16.954937 7022 raft_consensus.cc:2932] T 35c843bebb484f6cb655c146a02261ba P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:16.966390 9719 tablet_bootstrap.cc:492] T 0bf6bcf89d444b339612d731af604d0a P bceeae1c79964acabdae8ff771bf0c97: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:16.966943 9719 tablet_bootstrap.cc:492] T 0bf6bcf89d444b339612d731af604d0a P bceeae1c79964acabdae8ff771bf0c97: Bootstrap complete.
I20240501 20:10:16.967222 9719 ts_tablet_manager.cc:1367] T 0bf6bcf89d444b339612d731af604d0a P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.062s user 0.055s sys 0.004s
I20240501 20:10:16.968067 9719 raft_consensus.cc:348] T 0bf6bcf89d444b339612d731af604d0a P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:16.968413 9719 raft_consensus.cc:729] T 0bf6bcf89d444b339612d731af604d0a P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: LEARNER
I20240501 20:10:16.968621 9719 consensus_queue.cc:260] T 0bf6bcf89d444b339612d731af604d0a P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:16.969301 9719 ts_tablet_manager.cc:1398] T 0bf6bcf89d444b339612d731af604d0a P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:16.970181 4388 tablet_copy_service.cc:342] P 10b47782558c41ee91f5e164a372049a: Request end of tablet copy session bceeae1c79964acabdae8ff771bf0c97-0bf6bcf89d444b339612d731af604d0a received from {username='slave'} at 127.29.219.71:41437
I20240501 20:10:16.970326 4388 tablet_copy_service.cc:433] P 10b47782558c41ee91f5e164a372049a: ending tablet copy session bceeae1c79964acabdae8ff771bf0c97-0bf6bcf89d444b339612d731af604d0a on tablet 0bf6bcf89d444b339612d731af604d0a with peer bceeae1c79964acabdae8ff771bf0c97
I20240501 20:10:16.974478 7421 raft_consensus.cc:1203] T ddcb6f5090f143c2b672e5630230878f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:16.976569 9319 raft_consensus.cc:2932] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:16.979501 4365 raft_consensus.cc:2932] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:16.982185 4627 raft_consensus.cc:2932] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:16.984457 4093 catalog_manager.cc:5484] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:16.993835 9033 consensus_queue.cc:1035] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:16.993745 7285 raft_consensus.cc:2932] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:17.020287 9201 consensus_queue.cc:1035] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:17.029348 4365 consensus_queue.cc:237] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
W20240501 20:10:17.032212 4299 consensus_peers.cc:485] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a -> Peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Couldn't send request to peer db0742ee3860468590eefef6ae6074fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb5700eb47e044c182c33183ebb27908. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:17.032426 4499 raft_consensus.cc:1261] T bb5700eb47e044c182c33183ebb27908 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:17.032958 4233 raft_consensus.cc:1261] T bb5700eb47e044c182c33183ebb27908 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:17.032999 9552 consensus_queue.cc:1035] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:17.033576 9143 consensus_queue.cc:1035] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:17.035987 9552 raft_consensus.cc:2932] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:17.036695 4499 raft_consensus.cc:2932] T bb5700eb47e044c182c33183ebb27908 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:17.037135 4233 raft_consensus.cc:2932] T bb5700eb47e044c182c33183ebb27908 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:17.042681 4091 catalog_manager.cc:5484] T bb5700eb47e044c182c33183ebb27908 P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index -1 to 4, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:17.047873 4088 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:17.053974 7022 raft_consensus.cc:1261] T ca0fadee796b431f948266c96ab0c93d P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:17.053974 7153 raft_consensus.cc:1261] T ca0fadee796b431f948266c96ab0c93d P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:17.054571 9598 consensus_queue.cc:1035] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:17.055682 9598 consensus_queue.cc:1035] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:17.059433 4088 catalog_manager.cc:5576] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321) does not have the latest schema for tablet ca0fadee796b431f948266c96ab0c93d (table rebalancer_dynamic_table [id=1c0e2cb9c07f47a2b6505918ddbd35fc]). Expected version 1 got 0
I20240501 20:10:17.065337 4087 catalog_manager.cc:5576] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237) does not have the latest schema for tablet ca0fadee796b431f948266c96ab0c93d (table rebalancer_dynamic_table [id=1c0e2cb9c07f47a2b6505918ddbd35fc]). Expected version 1 got 0
I20240501 20:10:17.074594 8851 consensus_queue.cc:1035] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:17.074709 9767 tablet.cc:1722] T ca0fadee796b431f948266c96ab0c93d P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:17.078863 9765 tablet.cc:1722] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:17.079492 4087 catalog_manager.cc:5576] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237) does not have the latest schema for tablet ca0fadee796b431f948266c96ab0c93d (table rebalancer_dynamic_table [id=1c0e2cb9c07f47a2b6505918ddbd35fc]). Expected version 1 got 0
I20240501 20:10:17.083132 9765 tablet.cc:1620] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:17.083410 9767 tablet.cc:1620] T ca0fadee796b431f948266c96ab0c93d P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:17.084646 9766 tablet.cc:1722] T ca0fadee796b431f948266c96ab0c93d P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:17.087610 4088 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=1c0e2cb9c07f47a2b6505918ddbd35fc] alter complete (version 1)
I20240501 20:10:17.088593 9766 tablet.cc:1620] T ca0fadee796b431f948266c96ab0c93d P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:10:17.091691 4087 catalog_manager.cc:5576] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321) does not have the latest schema for tablet ca0fadee796b431f948266c96ab0c93d (table rebalancer_dynamic_table [id=1c0e2cb9c07f47a2b6505918ddbd35fc]). Expected version 1 got 0
I20240501 20:10:17.114383 8851 consensus_queue.cc:1035] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:17.192148 4088 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:17.204038 4499 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "65b0048212a745c08ab24ead346bf74e"
dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35098
I20240501 20:10:17.204293 4499 raft_consensus.cc:595] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Received request to transfer leadership
I20240501 20:10:17.208369 9765 tablet.cc:1722] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:17.208921 9765 tablet.cc:1620] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:17.209509 9767 tablet.cc:1722] T ca0fadee796b431f948266c96ab0c93d P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:17.209964 9767 tablet.cc:1620] T ca0fadee796b431f948266c96ab0c93d P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:17.210574 9766 tablet.cc:1722] T ca0fadee796b431f948266c96ab0c93d P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:17.211071 9766 tablet.cc:1620] T ca0fadee796b431f948266c96ab0c93d P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:10:17.212610 9158 consensus_queue.cc:1035] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:17.228534 7285 raft_consensus.cc:1203] T 0bf6bcf89d444b339612d731af604d0a P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:17.233558 4088 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=1c0e2cb9c07f47a2b6505918ddbd35fc] alter complete (version 2)
I20240501 20:10:17.234820 9533 consensus_queue.cc:1035] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:17.275485 4088 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:17.285634 7400 tablet_service.cc:1467] Processing CreateTablet for tablet 9c77905da1b3453fa9a83366f7c51c37 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=513576dc865743a18e99f37632b2a4a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:17.286170 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c77905da1b3453fa9a83366f7c51c37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:17.286193 7401 tablet_service.cc:1467] Processing CreateTablet for tablet c1e7ac38dc9c44d29e2d8127904407eb (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=513576dc865743a18e99f37632b2a4a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:17.286646 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1e7ac38dc9c44d29e2d8127904407eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:17.286638 7135 tablet_service.cc:1467] Processing CreateTablet for tablet c1e7ac38dc9c44d29e2d8127904407eb (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=513576dc865743a18e99f37632b2a4a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:17.287019 7135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1e7ac38dc9c44d29e2d8127904407eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:17.291643 4478 tablet_service.cc:1467] Processing CreateTablet for tablet 9c77905da1b3453fa9a83366f7c51c37 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=513576dc865743a18e99f37632b2a4a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:17.292109 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c77905da1b3453fa9a83366f7c51c37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:17.296136 7002 tablet_service.cc:1467] Processing CreateTablet for tablet 9c77905da1b3453fa9a83366f7c51c37 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=513576dc865743a18e99f37632b2a4a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:17.296619 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c77905da1b3453fa9a83366f7c51c37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:17.297186 7001 tablet_service.cc:1467] Processing CreateTablet for tablet 00ae88161b514e4aa4b8d134a19de001 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=513576dc865743a18e99f37632b2a4a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:17.297554 7001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ae88161b514e4aa4b8d134a19de001. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:17.299331 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 00ae88161b514e4aa4b8d134a19de001 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=513576dc865743a18e99f37632b2a4a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:17.299733 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ae88161b514e4aa4b8d134a19de001. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:17.301391 7267 tablet_service.cc:1467] Processing CreateTablet for tablet c1e7ac38dc9c44d29e2d8127904407eb (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=513576dc865743a18e99f37632b2a4a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:17.301739 4477 tablet_service.cc:1467] Processing CreateTablet for tablet 00ae88161b514e4aa4b8d134a19de001 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=513576dc865743a18e99f37632b2a4a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:17.301888 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1e7ac38dc9c44d29e2d8127904407eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:17.302165 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ae88161b514e4aa4b8d134a19de001. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:17.319207 9805 tablet_bootstrap.cc:492] T 00ae88161b514e4aa4b8d134a19de001 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:17.321697 9805 tablet_bootstrap.cc:654] T 00ae88161b514e4aa4b8d134a19de001 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:17.338925 9805 tablet_bootstrap.cc:492] T 00ae88161b514e4aa4b8d134a19de001 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:17.339294 9805 ts_tablet_manager.cc:1367] T 00ae88161b514e4aa4b8d134a19de001 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.009s
I20240501 20:10:17.340399 9805 raft_consensus.cc:348] T 00ae88161b514e4aa4b8d134a19de001 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:17.340816 9805 raft_consensus.cc:374] T 00ae88161b514e4aa4b8d134a19de001 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:17.340986 9805 raft_consensus.cc:729] T 00ae88161b514e4aa4b8d134a19de001 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:17.341373 9805 consensus_queue.cc:260] T 00ae88161b514e4aa4b8d134a19de001 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:17.342252 9805 ts_tablet_manager.cc:1398] T 00ae88161b514e4aa4b8d134a19de001 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:17.342346 9809 tablet_bootstrap.cc:492] T 9c77905da1b3453fa9a83366f7c51c37 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:17.342824 9812 tablet_bootstrap.cc:492] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:17.343416 9805 tablet_bootstrap.cc:492] T 9c77905da1b3453fa9a83366f7c51c37 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:17.344700 9809 tablet_bootstrap.cc:654] T 9c77905da1b3453fa9a83366f7c51c37 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:17.345481 9805 tablet_bootstrap.cc:654] T 9c77905da1b3453fa9a83366f7c51c37 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:17.347522 4093 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:17.348666 9812 tablet_bootstrap.cc:654] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:17.350838 9807 tablet_bootstrap.cc:492] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:17.351445 9808 tablet_bootstrap.cc:492] T c1e7ac38dc9c44d29e2d8127904407eb P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:17.358286 9811 ts_tablet_manager.cc:902] T 35c843bebb484f6cb655c146a02261ba P 3e974c03d3c546f9ab6c91e762a5426b: Initiating tablet copy from peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:17.359879 9805 tablet_bootstrap.cc:492] T 9c77905da1b3453fa9a83366f7c51c37 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:17.360172 9805 ts_tablet_manager.cc:1367] T 9c77905da1b3453fa9a83366f7c51c37 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.003s
I20240501 20:10:17.360230 9807 tablet_bootstrap.cc:654] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:17.361008 9808 tablet_bootstrap.cc:654] T c1e7ac38dc9c44d29e2d8127904407eb P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:17.361105 9805 raft_consensus.cc:348] T 9c77905da1b3453fa9a83366f7c51c37 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:17.361353 9805 raft_consensus.cc:374] T 9c77905da1b3453fa9a83366f7c51c37 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:17.361433 9805 raft_consensus.cc:729] T 9c77905da1b3453fa9a83366f7c51c37 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:17.361723 9805 consensus_queue.cc:260] T 9c77905da1b3453fa9a83366f7c51c37 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:17.362254 9805 ts_tablet_manager.cc:1398] T 9c77905da1b3453fa9a83366f7c51c37 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:17.365422 9765 tablet.cc:1722] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:17.366793 9804 tablet_bootstrap.cc:492] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:17.367604 9809 tablet_bootstrap.cc:492] T 9c77905da1b3453fa9a83366f7c51c37 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:10:17.368009 9809 ts_tablet_manager.cc:1367] T 9c77905da1b3453fa9a83366f7c51c37 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.000s
W20240501 20:10:17.370368 4563 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.71:39447, user_credentials={real_user=slave}} blocked reactor thread for 58181.5us
I20240501 20:10:17.371423 9767 tablet.cc:1722] T ca0fadee796b431f948266c96ab0c93d P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:17.371542 9809 raft_consensus.cc:348] T 9c77905da1b3453fa9a83366f7c51c37 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:17.371791 9809 raft_consensus.cc:374] T 9c77905da1b3453fa9a83366f7c51c37 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:17.371905 9809 raft_consensus.cc:729] T 9c77905da1b3453fa9a83366f7c51c37 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:10:17.373136 9812 tablet_bootstrap.cc:492] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:17.373358 9812 ts_tablet_manager.cc:1367] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.007s
I20240501 20:10:17.374166 9812 raft_consensus.cc:348] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:17.374373 9812 raft_consensus.cc:374] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:17.374436 9812 raft_consensus.cc:729] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:17.374668 9812 consensus_queue.cc:260] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:17.375178 9812 ts_tablet_manager.cc:1398] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:17.375687 9809 consensus_queue.cc:260] T 9c77905da1b3453fa9a83366f7c51c37 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:17.377820 9804 tablet_bootstrap.cc:654] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:17.381750 9809 ts_tablet_manager.cc:1398] T 9c77905da1b3453fa9a83366f7c51c37 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240501 20:10:17.382146 9809 tablet_bootstrap.cc:492] T 00ae88161b514e4aa4b8d134a19de001 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:17.383273 9319 raft_consensus.cc:979] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b: : Instructing follower 1d34e18a1d394d3f91c9f50d1c7bea4e to start an election
I20240501 20:10:17.383450 9319 raft_consensus.cc:1067] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Signalling peer 1d34e18a1d394d3f91c9f50d1c7bea4e to start an election
I20240501 20:10:17.384311 9809 tablet_bootstrap.cc:654] T 00ae88161b514e4aa4b8d134a19de001 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:17.385167 9811 tablet_copy_client.cc:308] T 35c843bebb484f6cb655c146a02261ba P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.70:36321
I20240501 20:10:17.388149 9807 tablet_bootstrap.cc:492] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:17.388177 7175 tablet_copy_service.cc:140] P db0742ee3860468590eefef6ae6074fd: Received BeginTabletCopySession request for tablet 35c843bebb484f6cb655c146a02261ba from peer 3e974c03d3c546f9ab6c91e762a5426b ({username='slave'} at 127.29.219.67:45693)
I20240501 20:10:17.388358 9807 ts_tablet_manager.cc:1367] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.002s
I20240501 20:10:17.388403 7175 tablet_copy_service.cc:161] P db0742ee3860468590eefef6ae6074fd: Beginning new tablet copy session on tablet 35c843bebb484f6cb655c146a02261ba from peer 3e974c03d3c546f9ab6c91e762a5426b at {username='slave'} at 127.29.219.67:45693: session id = 3e974c03d3c546f9ab6c91e762a5426b-35c843bebb484f6cb655c146a02261ba
I20240501 20:10:17.389349 9807 raft_consensus.cc:348] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:17.389580 9807 raft_consensus.cc:374] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:17.389714 9807 raft_consensus.cc:729] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:17.390060 9807 consensus_queue.cc:260] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:17.390709 9807 ts_tablet_manager.cc:1398] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:17.391017 4631 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65b0048212a745c08ab24ead346bf74e"
dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
from {username='slave'} at 127.29.219.67:55693
I20240501 20:10:17.391369 4631 raft_consensus.cc:482] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:10:17.391587 4631 raft_consensus.cc:3037] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:17.392131 9766 tablet.cc:1722] T ca0fadee796b431f948266c96ab0c93d P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:17.401221 9720 raft_consensus.cc:482] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:17.401388 9720 raft_consensus.cc:504] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:17.402231 9720 leader_election.cc:288] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:17.402261 9808 tablet_bootstrap.cc:492] T c1e7ac38dc9c44d29e2d8127904407eb P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:17.402447 9808 ts_tablet_manager.cc:1367] T c1e7ac38dc9c44d29e2d8127904407eb P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.005s
I20240501 20:10:17.402523 4093 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table_new_name [id=1c0e2cb9c07f47a2b6505918ddbd35fc] alter complete (version 3)
I20240501 20:10:17.402972 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e7ac38dc9c44d29e2d8127904407eb" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:17.403456 9808 raft_consensus.cc:348] T c1e7ac38dc9c44d29e2d8127904407eb P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
W20240501 20:10:17.403690 7223 leader_election.cc:341] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:17.403870 9808 raft_consensus.cc:374] T c1e7ac38dc9c44d29e2d8127904407eb P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:17.404029 9808 raft_consensus.cc:729] T c1e7ac38dc9c44d29e2d8127904407eb P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:17.404345 9808 consensus_queue.cc:260] T c1e7ac38dc9c44d29e2d8127904407eb P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:17.404768 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e7ac38dc9c44d29e2d8127904407eb" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:17.405025 7153 raft_consensus.cc:2454] T c1e7ac38dc9c44d29e2d8127904407eb P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:17.405339 9808 ts_tablet_manager.cc:1398] T c1e7ac38dc9c44d29e2d8127904407eb P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:10:17.405355 7220 leader_election.cc:302] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, db0742ee3860468590eefef6ae6074fd; no voters: b9957d4086bc4a76b18ed9265a5739d5
I20240501 20:10:17.405649 9720 raft_consensus.cc:2781] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:17.405777 9720 raft_consensus.cc:482] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:17.405879 9720 raft_consensus.cc:3037] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:17.407140 4631 raft_consensus.cc:504] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:17.408542 9720 raft_consensus.cc:504] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:17.409583 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e7ac38dc9c44d29e2d8127904407eb" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:17.409991 9720 leader_election.cc:288] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Requested vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
W20240501 20:10:17.410599 7223 leader_election.cc:341] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:17.410954 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e7ac38dc9c44d29e2d8127904407eb" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:17.411231 7153 raft_consensus.cc:3037] T c1e7ac38dc9c44d29e2d8127904407eb P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:17.413106 7175 tablet_copy_source_session.cc:205] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:17.416023 9811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35c843bebb484f6cb655c146a02261ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:17.418002 7153 raft_consensus.cc:2454] T c1e7ac38dc9c44d29e2d8127904407eb P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:17.418546 7220 leader_election.cc:302] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, db0742ee3860468590eefef6ae6074fd; no voters: b9957d4086bc4a76b18ed9265a5739d5
I20240501 20:10:17.418448 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b0048212a745c08ab24ead346bf74e" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:17.419003 7285 raft_consensus.cc:3037] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:17.420138 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b0048212a745c08ab24ead346bf74e" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:10:17.420356 4365 raft_consensus.cc:3037] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:17.420372 9720 raft_consensus.cc:2781] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:17.420554 9720 raft_consensus.cc:686] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Becoming Leader. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Running, Role: LEADER
I20240501 20:10:17.420840 9720 consensus_queue.cc:237] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:17.428707 7285 raft_consensus.cc:2454] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 2.
I20240501 20:10:17.417729 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b0048212a745c08ab24ead346bf74e" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:10:17.429970 4499 raft_consensus.cc:3032] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:10:17.430205 4499 raft_consensus.cc:729] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:10:17.430640 4499 consensus_queue.cc:260] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:17.431324 4499 raft_consensus.cc:3037] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:17.434670 4365 raft_consensus.cc:2454] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 2.
I20240501 20:10:17.435341 4564 leader_election.cc:302] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 1d34e18a1d394d3f91c9f50d1c7bea4e, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:17.435822 9555 raft_consensus.cc:2781] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:17.436038 9555 raft_consensus.cc:686] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 LEADER]: Becoming Leader. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Running, Role: LEADER
I20240501 20:10:17.436475 9555 consensus_queue.cc:237] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:17.444835 4088 catalog_manager.cc:5484] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: term changed from 1 to 2, leader changed from 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) to 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68). New cstate: current_term: 2 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:17.457036 4631 leader_election.cc:288] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:17.467643 9809 tablet_bootstrap.cc:492] T 00ae88161b514e4aa4b8d134a19de001 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:10:17.468045 9809 ts_tablet_manager.cc:1367] T 00ae88161b514e4aa4b8d134a19de001 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.086s user 0.004s sys 0.000s
I20240501 20:10:17.469056 9809 raft_consensus.cc:348] T 00ae88161b514e4aa4b8d134a19de001 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:17.470271 9809 raft_consensus.cc:374] T 00ae88161b514e4aa4b8d134a19de001 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:17.470260 9802 raft_consensus.cc:482] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:17.470522 9809 raft_consensus.cc:729] T 00ae88161b514e4aa4b8d134a19de001 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:10:17.470398 9802 raft_consensus.cc:504] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:17.471021 9809 consensus_queue.cc:260] T 00ae88161b514e4aa4b8d134a19de001 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:17.471237 9802 leader_election.cc:288] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:17.472041 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ae88161b514e4aa4b8d134a19de001" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:17.472318 7022 raft_consensus.cc:2454] T 00ae88161b514e4aa4b8d134a19de001 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:17.472407 9804 tablet_bootstrap.cc:492] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:17.472609 9804 ts_tablet_manager.cc:1367] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.106s user 0.005s sys 0.000s
I20240501 20:10:17.473006 4496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ae88161b514e4aa4b8d134a19de001" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:17.473193 4496 raft_consensus.cc:2454] T 00ae88161b514e4aa4b8d134a19de001 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:17.473435 9809 ts_tablet_manager.cc:1398] T 00ae88161b514e4aa4b8d134a19de001 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:10:17.473948 4169 leader_election.cc:302] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:17.473847 9804 raft_consensus.cc:348] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:17.474341 9804 raft_consensus.cc:374] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:17.474516 9804 raft_consensus.cc:729] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:17.474975 9804 consensus_queue.cc:260] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:17.477263 9804 ts_tablet_manager.cc:1398] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:10:17.477663 9804 tablet_bootstrap.cc:492] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:17.478520 9802 raft_consensus.cc:2781] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:17.478668 9802 raft_consensus.cc:482] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:17.478744 9802 raft_consensus.cc:3037] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:17.481300 9811 tablet_copy_client.cc:783] T 35c843bebb484f6cb655c146a02261ba P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:17.481673 9811 tablet_copy_client.cc:647] T 35c843bebb484f6cb655c146a02261ba P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:17.482378 9802 raft_consensus.cc:504] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:17.483703 9811 tablet_copy_client.cc:523] T 35c843bebb484f6cb655c146a02261ba P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:17.483894 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ae88161b514e4aa4b8d134a19de001" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:17.484270 7022 raft_consensus.cc:3037] T 00ae88161b514e4aa4b8d134a19de001 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:17.485822 9804 tablet_bootstrap.cc:654] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:17.490847 4499 raft_consensus.cc:2454] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 2.
I20240501 20:10:17.497712 9811 tablet_bootstrap.cc:492] T 35c843bebb484f6cb655c146a02261ba P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:17.500389 7022 raft_consensus.cc:2454] T 00ae88161b514e4aa4b8d134a19de001 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:17.500823 4169 leader_election.cc:302] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dca57823003e4729b769ca12367d7530, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:17.501078 8785 raft_consensus.cc:2781] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:17.501317 4088 catalog_manager.cc:5484] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71). New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:17.502959 9802 leader_election.cc:288] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:17.504408 8785 raft_consensus.cc:686] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:10:17.504803 9660 raft_consensus.cc:482] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:17.504760 8785 consensus_queue.cc:237] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:17.504984 9660 raft_consensus.cc:504] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:17.505117 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ae88161b514e4aa4b8d134a19de001" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:10:17.505312 4499 raft_consensus.cc:3037] T 00ae88161b514e4aa4b8d134a19de001 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:17.505623 9660 leader_election.cc:288] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:17.506695 4496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c77905da1b3453fa9a83366f7c51c37" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:17.506980 4496 raft_consensus.cc:2454] T 9c77905da1b3453fa9a83366f7c51c37 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:10:17.507759 7355 leader_election.cc:302] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, b9957d4086bc4a76b18ed9265a5739d5; no voters:
I20240501 20:10:17.509176 9804 tablet_bootstrap.cc:492] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:17.509580 9804 ts_tablet_manager.cc:1367] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.000s
I20240501 20:10:17.509526 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c77905da1b3453fa9a83366f7c51c37" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:17.509884 7022 raft_consensus.cc:2454] T 9c77905da1b3453fa9a83366f7c51c37 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:10:17.510727 4088 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:17.510977 4088 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:17.513126 9660 raft_consensus.cc:2781] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:17.513238 9660 raft_consensus.cc:482] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:17.513346 9660 raft_consensus.cc:3037] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:17.512048 9804 raft_consensus.cc:348] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:17.513970 9804 raft_consensus.cc:374] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:17.514102 9804 raft_consensus.cc:729] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:17.514526 9804 consensus_queue.cc:260] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:17.518702 4088 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet ca0fadee796b431f948266c96ab0c93d
I20240501 20:10:17.517256 9804 ts_tablet_manager.cc:1398] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 20:10:17.519964 7135 tablet_service.cc:1514] Processing DeleteTablet for tablet ca0fadee796b431f948266c96ab0c93d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:17 UTC) from {username='slave'} at 127.0.0.1:52036
I20240501 20:10:17.521306 9832 tablet_replica.cc:330] T ca0fadee796b431f948266c96ab0c93d P db0742ee3860468590eefef6ae6074fd: stopping tablet replica
I20240501 20:10:17.521735 7267 tablet_service.cc:1514] Processing DeleteTablet for tablet ca0fadee796b431f948266c96ab0c93d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:17 UTC) from {username='slave'} at 127.0.0.1:54166
I20240501 20:10:17.522222 9832 raft_consensus.cc:2229] T ca0fadee796b431f948266c96ab0c93d P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:17.522480 9832 raft_consensus.cc:2258] T ca0fadee796b431f948266c96ab0c93d P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:17.524076 9832 ts_tablet_manager.cc:1875] T ca0fadee796b431f948266c96ab0c93d P db0742ee3860468590eefef6ae6074fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:17.526330 7002 tablet_service.cc:1514] Processing DeleteTablet for tablet ca0fadee796b431f948266c96ab0c93d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:17 UTC) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:17.531785 9835 ts_tablet_manager.cc:902] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd: Initiating tablet copy from peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:17.533329 9660 raft_consensus.cc:504] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:17.534448 4499 raft_consensus.cc:2454] T 00ae88161b514e4aa4b8d134a19de001 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:17.535192 9834 tablet_replica.cc:330] T ca0fadee796b431f948266c96ab0c93d P ef0a4ada548446b3a531cb39a3c4e38d: stopping tablet replica
I20240501 20:10:17.536520 9833 tablet_replica.cc:330] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:17.536892 9833 raft_consensus.cc:2229] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:17.537171 9660 leader_election.cc:288] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:17.537422 9833 raft_consensus.cc:2258] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:17.538866 9833 ts_tablet_manager.cc:1875] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:17.540585 9834 raft_consensus.cc:2229] T ca0fadee796b431f948266c96ab0c93d P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:17.540927 9834 raft_consensus.cc:2258] T ca0fadee796b431f948266c96ab0c93d P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:17.536415 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c77905da1b3453fa9a83366f7c51c37" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:17.541477 7022 raft_consensus.cc:3037] T 9c77905da1b3453fa9a83366f7c51c37 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:17.544891 9834 ts_tablet_manager.cc:1875] T ca0fadee796b431f948266c96ab0c93d P ef0a4ada548446b3a531cb39a3c4e38d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:17.545248 7022 raft_consensus.cc:2454] T 9c77905da1b3453fa9a83366f7c51c37 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:17.545795 7355 leader_election.cc:302] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:17.550132 9833 ts_tablet_manager.cc:1888] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:17.550360 9660 raft_consensus.cc:2781] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:17.550590 9660 raft_consensus.cc:686] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Becoming Leader. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Running, Role: LEADER
I20240501 20:10:17.550455 9833 log.cc:1196] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/ca0fadee796b431f948266c96ab0c93d
I20240501 20:10:17.551863 9833 ts_tablet_manager.cc:1909] T ca0fadee796b431f948266c96ab0c93d P bceeae1c79964acabdae8ff771bf0c97: Deleting consensus metadata
I20240501 20:10:17.550968 9660 consensus_queue.cc:237] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:17.553386 9835 tablet_copy_client.cc:308] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.66:40689
I20240501 20:10:17.553583 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c77905da1b3453fa9a83366f7c51c37" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:10:17.553776 4499 raft_consensus.cc:3037] T 9c77905da1b3453fa9a83366f7c51c37 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:17.554198 4388 tablet_copy_service.cc:140] P 10b47782558c41ee91f5e164a372049a: Received BeginTabletCopySession request for tablet bb5700eb47e044c182c33183ebb27908 from peer db0742ee3860468590eefef6ae6074fd ({username='slave'} at 127.29.219.70:42453)
I20240501 20:10:17.554364 4388 tablet_copy_service.cc:161] P 10b47782558c41ee91f5e164a372049a: Beginning new tablet copy session on tablet bb5700eb47e044c182c33183ebb27908 from peer db0742ee3860468590eefef6ae6074fd at {username='slave'} at 127.29.219.70:42453: session id = db0742ee3860468590eefef6ae6074fd-bb5700eb47e044c182c33183ebb27908
I20240501 20:10:17.556238 9811 tablet_bootstrap.cc:492] T 35c843bebb484f6cb655c146a02261ba P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:17.557448 9811 tablet_bootstrap.cc:492] T 35c843bebb484f6cb655c146a02261ba P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap complete.
I20240501 20:10:17.557451 4388 tablet_copy_source_session.cc:205] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:17.557996 9811 ts_tablet_manager.cc:1367] T 35c843bebb484f6cb655c146a02261ba P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.060s user 0.009s sys 0.000s
I20240501 20:10:17.559302 9811 raft_consensus.cc:348] T 35c843bebb484f6cb655c146a02261ba P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:17.559847 9811 raft_consensus.cc:729] T 35c843bebb484f6cb655c146a02261ba P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: LEARNER
I20240501 20:10:17.560185 9811 consensus_queue.cc:260] T 35c843bebb484f6cb655c146a02261ba P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:17.561242 4499 raft_consensus.cc:2454] T 9c77905da1b3453fa9a83366f7c51c37 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:17.563752 9835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb5700eb47e044c182c33183ebb27908. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:17.566145 9811 ts_tablet_manager.cc:1398] T 35c843bebb484f6cb655c146a02261ba P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 20:10:17.570066 9143 raft_consensus.cc:1050] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a: attempting to promote NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 to VOTER
I20240501 20:10:17.570165 4091 catalog_manager.cc:5484] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:17.570748 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet ca0fadee796b431f948266c96ab0c93d (table rebalancer_dynamic_table_new_name [id=1c0e2cb9c07f47a2b6505918ddbd35fc]) successfully deleted
I20240501 20:10:17.571035 9143 consensus_queue.cc:237] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:17.572592 7175 tablet_copy_service.cc:342] P db0742ee3860468590eefef6ae6074fd: Request end of tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-35c843bebb484f6cb655c146a02261ba received from {username='slave'} at 127.29.219.67:45693
I20240501 20:10:17.572644 9832 ts_tablet_manager.cc:1888] T ca0fadee796b431f948266c96ab0c93d P db0742ee3860468590eefef6ae6074fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:17.572751 7175 tablet_copy_service.cc:433] P db0742ee3860468590eefef6ae6074fd: ending tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-35c843bebb484f6cb655c146a02261ba on tablet 35c843bebb484f6cb655c146a02261ba with peer 3e974c03d3c546f9ab6c91e762a5426b
I20240501 20:10:17.572784 9832 log.cc:1196] T ca0fadee796b431f948266c96ab0c93d P db0742ee3860468590eefef6ae6074fd: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/wals/ca0fadee796b431f948266c96ab0c93d
I20240501 20:10:17.579149 9832 ts_tablet_manager.cc:1909] T ca0fadee796b431f948266c96ab0c93d P db0742ee3860468590eefef6ae6074fd: Deleting consensus metadata
I20240501 20:10:17.581102 4076 catalog_manager.cc:4830] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): tablet ca0fadee796b431f948266c96ab0c93d (table rebalancer_dynamic_table_new_name [id=1c0e2cb9c07f47a2b6505918ddbd35fc]) successfully deleted
I20240501 20:10:17.584254 9834 ts_tablet_manager.cc:1888] T ca0fadee796b431f948266c96ab0c93d P ef0a4ada548446b3a531cb39a3c4e38d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:17.584010 4091 catalog_manager.cc:5484] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72). New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:17.584584 9834 log.cc:1196] T ca0fadee796b431f948266c96ab0c93d P ef0a4ada548446b3a531cb39a3c4e38d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/wals/ca0fadee796b431f948266c96ab0c93d
I20240501 20:10:17.585125 9835 tablet_copy_client.cc:783] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:17.585412 9835 tablet_copy_client.cc:647] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:17.585616 9834 ts_tablet_manager.cc:1909] T ca0fadee796b431f948266c96ab0c93d P ef0a4ada548446b3a531cb39a3c4e38d: Deleting consensus metadata
I20240501 20:10:17.587030 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet ca0fadee796b431f948266c96ab0c93d (table rebalancer_dynamic_table_new_name [id=1c0e2cb9c07f47a2b6505918ddbd35fc]) successfully deleted
I20240501 20:10:17.588426 7419 raft_consensus.cc:1261] T ddcb6f5090f143c2b672e5630230878f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:17.589390 9552 consensus_queue.cc:1035] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:17.589783 4631 raft_consensus.cc:1261] T ddcb6f5090f143c2b672e5630230878f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:17.590571 9552 consensus_queue.cc:1035] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:17.590797 4499 raft_consensus.cc:1261] T ddcb6f5090f143c2b672e5630230878f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:17.591339 9552 consensus_queue.cc:1035] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:17.598043 9835 tablet_copy_client.cc:523] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:17.607904 9835 tablet_bootstrap.cc:492] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:17.616549 9552 raft_consensus.cc:2932] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:17.621340 7419 raft_consensus.cc:2932] T ddcb6f5090f143c2b672e5630230878f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:17.623090 4091 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:17.630653 4496 raft_consensus.cc:2932] T ddcb6f5090f143c2b672e5630230878f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:17.631052 4088 catalog_manager.cc:5484] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:17.633458 4627 raft_consensus.cc:2932] T ddcb6f5090f143c2b672e5630230878f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:17.637080 4346 tablet_service.cc:1467] Processing CreateTablet for tablet bc87c8f97be84c05adad5e13ab5228b3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3696b5f0f4bb4fffa1e15ccf7bc4f8c8]), partition=
I20240501 20:10:17.637511 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc87c8f97be84c05adad5e13ab5228b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:17.640995 7400 tablet_service.cc:1467] Processing CreateTablet for tablet bc87c8f97be84c05adad5e13ab5228b3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3696b5f0f4bb4fffa1e15ccf7bc4f8c8]), partition=
I20240501 20:10:17.641366 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc87c8f97be84c05adad5e13ab5228b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:17.648550 7267 tablet_service.cc:1467] Processing CreateTablet for tablet bc87c8f97be84c05adad5e13ab5228b3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3696b5f0f4bb4fffa1e15ccf7bc4f8c8]), partition=
I20240501 20:10:17.649011 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc87c8f97be84c05adad5e13ab5228b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:17.653805 4365 consensus_queue.cc:237] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:17.654832 9804 tablet_bootstrap.cc:492] T bc87c8f97be84c05adad5e13ab5228b3 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:17.656533 9804 tablet_bootstrap.cc:654] T bc87c8f97be84c05adad5e13ab5228b3 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:17.657298 7419 raft_consensus.cc:1261] T ddcb6f5090f143c2b672e5630230878f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:17.657877 9744 consensus_queue.cc:1035] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:17.657496 4627 raft_consensus.cc:1261] T ddcb6f5090f143c2b672e5630230878f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:17.659070 9744 consensus_queue.cc:1035] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:17.662168 9744 raft_consensus.cc:2932] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:17.664805 9807 tablet_bootstrap.cc:492] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:17.665478 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ddcb6f5090f143c2b672e5630230878f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:17.666476 9807 tablet_bootstrap.cc:654] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:17.666935 7419 raft_consensus.cc:2932] T ddcb6f5090f143c2b672e5630230878f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:17.671253 4631 raft_consensus.cc:2932] T ddcb6f5090f143c2b672e5630230878f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:17.681764 4092 catalog_manager.cc:5484] T ddcb6f5090f143c2b672e5630230878f P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:17.685252 9807 tablet_bootstrap.cc:492] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:17.685503 9807 ts_tablet_manager.cc:1367] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.004s
I20240501 20:10:17.686558 9807 raft_consensus.cc:348] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:17.687064 9807 raft_consensus.cc:374] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:17.687832 9807 raft_consensus.cc:729] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:17.692039 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet ddcb6f5090f143c2b672e5630230878f with delete_type TABLET_DATA_TOMBSTONED (TS 3e974c03d3c546f9ab6c91e762a5426b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:17.712430 9804 tablet_bootstrap.cc:492] T bc87c8f97be84c05adad5e13ab5228b3 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:17.713006 9804 ts_tablet_manager.cc:1367] T bc87c8f97be84c05adad5e13ab5228b3 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.058s user 0.001s sys 0.011s
I20240501 20:10:17.715620 9804 raft_consensus.cc:348] T bc87c8f97be84c05adad5e13ab5228b3 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:17.715744 4499 raft_consensus.cc:1203] T 35c843bebb484f6cb655c146a02261ba P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:17.716563 9804 raft_consensus.cc:374] T bc87c8f97be84c05adad5e13ab5228b3 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:17.716755 9804 raft_consensus.cc:729] T bc87c8f97be84c05adad5e13ab5228b3 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:17.719143 9552 raft_consensus.cc:1050] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a: attempting to promote NON_VOTER bceeae1c79964acabdae8ff771bf0c97 to VOTER
I20240501 20:10:17.718891 9804 consensus_queue.cc:260] T bc87c8f97be84c05adad5e13ab5228b3 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:17.720008 9552 consensus_queue.cc:237] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:17.734056 9804 ts_tablet_manager.cc:1398] T bc87c8f97be84c05adad5e13ab5228b3 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20240501 20:10:17.737895 9807 consensus_queue.cc:260] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:17.740782 9849 tablet_bootstrap.cc:492] T bc87c8f97be84c05adad5e13ab5228b3 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:17.742705 9849 tablet_bootstrap.cc:654] T bc87c8f97be84c05adad5e13ab5228b3 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:17.745208 9807 ts_tablet_manager.cc:1398] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.060s user 0.003s sys 0.000s
I20240501 20:10:17.752987 7285 raft_consensus.cc:1261] T 0bf6bcf89d444b339612d731af604d0a P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:17.753731 9744 consensus_queue.cc:1035] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:17.765676 9855 tablet_replica.cc:330] T ddcb6f5090f143c2b672e5630230878f P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:17.766072 9855 raft_consensus.cc:2229] T ddcb6f5090f143c2b672e5630230878f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:17.766408 9855 raft_consensus.cc:2258] T ddcb6f5090f143c2b672e5630230878f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:17.768798 9855 ts_tablet_manager.cc:1875] T ddcb6f5090f143c2b672e5630230878f P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:17.774113 4233 raft_consensus.cc:1261] T 0bf6bcf89d444b339612d731af604d0a P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:17.774931 9744 consensus_queue.cc:1035] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:17.775400 9849 tablet_bootstrap.cc:492] T bc87c8f97be84c05adad5e13ab5228b3 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:10:17.775570 9849 ts_tablet_manager.cc:1367] T bc87c8f97be84c05adad5e13ab5228b3 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.003s
I20240501 20:10:17.776504 9849 raft_consensus.cc:348] T bc87c8f97be84c05adad5e13ab5228b3 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:17.776837 9849 raft_consensus.cc:374] T bc87c8f97be84c05adad5e13ab5228b3 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:17.777079 9849 raft_consensus.cc:729] T bc87c8f97be84c05adad5e13ab5228b3 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:10:17.777513 9854 raft_consensus.cc:482] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:17.777750 9854 raft_consensus.cc:504] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:17.778821 9854 leader_election.cc:288] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:17.780968 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e7ac38dc9c44d29e2d8127904407eb" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:17.781293 7153 raft_consensus.cc:2379] T c1e7ac38dc9c44d29e2d8127904407eb P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9957d4086bc4a76b18ed9265a5739d5 in current term 1: Already voted for candidate bceeae1c79964acabdae8ff771bf0c97 in this term.
I20240501 20:10:17.781476 9849 consensus_queue.cc:260] T bc87c8f97be84c05adad5e13ab5228b3 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:17.783524 9849 ts_tablet_manager.cc:1398] T bc87c8f97be84c05adad5e13ab5228b3 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20240501 20:10:17.783712 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e7ac38dc9c44d29e2d8127904407eb" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:17.784375 7354 leader_election.cc:302] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5; no voters: bceeae1c79964acabdae8ff771bf0c97, db0742ee3860468590eefef6ae6074fd
I20240501 20:10:17.785512 4496 raft_consensus.cc:1261] T 0bf6bcf89d444b339612d731af604d0a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:17.786351 9854 raft_consensus.cc:3037] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:17.791585 9854 raft_consensus.cc:2726] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:17.797515 9552 consensus_queue.cc:1035] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:17.801021 9744 raft_consensus.cc:2932] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:17.806499 9855 ts_tablet_manager.cc:1888] T ddcb6f5090f143c2b672e5630230878f P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:17.807233 9855 log.cc:1196] T ddcb6f5090f143c2b672e5630230878f P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/ddcb6f5090f143c2b672e5630230878f
I20240501 20:10:17.806828 4233 raft_consensus.cc:2932] T 0bf6bcf89d444b339612d731af604d0a P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:17.809837 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet ddcb6f5090f143c2b672e5630230878f (table rebalance_test_table_5 [id=1e137c7380c64655a3b011211c536d72]) successfully deleted
I20240501 20:10:17.812485 4496 raft_consensus.cc:2932] T 0bf6bcf89d444b339612d731af604d0a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:17.816358 4092 catalog_manager.cc:5484] T 0bf6bcf89d444b339612d731af604d0a P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:17.824170 7285 raft_consensus.cc:2932] T 0bf6bcf89d444b339612d731af604d0a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:17.834028 4365 raft_consensus.cc:1261] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a [term 2 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:10:17.834609 9555 consensus_queue.cc:1035] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:17.851204 4498 raft_consensus.cc:1261] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:10:17.852020 9555 consensus_queue.cc:1035] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:17.852336 4365 consensus_queue.cc:237] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:17.856269 4233 raft_consensus.cc:1261] T 0bf6bcf89d444b339612d731af604d0a P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:17.857307 7285 raft_consensus.cc:1261] T 0bf6bcf89d444b339612d731af604d0a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:17.860235 9552 consensus_queue.cc:1035] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:17.860757 9835 tablet_bootstrap.cc:492] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:17.861215 9835 tablet_bootstrap.cc:492] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd: Bootstrap complete.
I20240501 20:10:17.861611 9835 ts_tablet_manager.cc:1367] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.254s user 0.064s sys 0.009s
I20240501 20:10:17.862468 9835 raft_consensus.cc:348] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:17.862836 9835 raft_consensus.cc:729] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: LEARNER
I20240501 20:10:17.863087 9835 consensus_queue.cc:260] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:17.864944 9744 consensus_queue.cc:1035] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:17.868597 9835 ts_tablet_manager.cc:1398] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 20:10:17.870564 9552 raft_consensus.cc:2932] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:17.873050 4388 tablet_copy_service.cc:342] P 10b47782558c41ee91f5e164a372049a: Request end of tablet copy session db0742ee3860468590eefef6ae6074fd-bb5700eb47e044c182c33183ebb27908 received from {username='slave'} at 127.29.219.70:42453
I20240501 20:10:17.873204 4388 tablet_copy_service.cc:433] P 10b47782558c41ee91f5e164a372049a: ending tablet copy session db0742ee3860468590eefef6ae6074fd-bb5700eb47e044c182c33183ebb27908 on tablet bb5700eb47e044c182c33183ebb27908 with peer db0742ee3860468590eefef6ae6074fd
I20240501 20:10:17.875849 4233 raft_consensus.cc:2932] T 0bf6bcf89d444b339612d731af604d0a P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:17.878243 7285 raft_consensus.cc:2932] T 0bf6bcf89d444b339612d731af604d0a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:17.886261 4092 catalog_manager.cc:5484] T 0bf6bcf89d444b339612d731af604d0a P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:17.890275 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bf6bcf89d444b339612d731af604d0a with delete_type TABLET_DATA_TOMBSTONED (TS 3e974c03d3c546f9ab6c91e762a5426b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:17.890802 9855 tablet_replica.cc:330] T 0bf6bcf89d444b339612d731af604d0a P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:17.891079 9855 raft_consensus.cc:2229] T 0bf6bcf89d444b339612d731af604d0a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:17.891314 9855 raft_consensus.cc:2258] T 0bf6bcf89d444b339612d731af604d0a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:17.892177 9855 ts_tablet_manager.cc:1875] T 0bf6bcf89d444b339612d731af604d0a P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:17.897464 9855 ts_tablet_manager.cc:1888] T 0bf6bcf89d444b339612d731af604d0a P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:17.897660 9855 log.cc:1196] T 0bf6bcf89d444b339612d731af604d0a P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/0bf6bcf89d444b339612d731af604d0a
I20240501 20:10:17.900543 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0bf6bcf89d444b339612d731af604d0a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:17.903709 7153 raft_consensus.cc:1203] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 20:10:17.906055 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet 0bf6bcf89d444b339612d731af604d0a (table rebalance_test_table_1 [id=eccfbde58fa5455b9f89a20ad3a37f39]) successfully deleted
I20240501 20:10:17.920610 9829 consensus_queue.cc:1035] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:17.932428 7285 raft_consensus.cc:1261] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:10:17.933378 9806 consensus_queue.cc:1035] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:17.938375 9720 consensus_queue.cc:1035] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:17.939561 9829 consensus_queue.cc:1035] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:17.975814 9598 consensus_queue.cc:1035] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:17.987140 9854 consensus_queue.cc:1035] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:18.012624 8851 consensus_queue.cc:1035] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:18.121356 9720 raft_consensus.cc:482] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:18.121548 9720 raft_consensus.cc:504] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:18.122368 9720 leader_election.cc:288] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:18.122845 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc87c8f97be84c05adad5e13ab5228b3" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:10:18.123070 4365 raft_consensus.cc:2454] T bc87c8f97be84c05adad5e13ab5228b3 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:18.123098 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc87c8f97be84c05adad5e13ab5228b3" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:18.123329 7420 raft_consensus.cc:2454] T bc87c8f97be84c05adad5e13ab5228b3 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:18.123685 7222 leader_election.cc:302] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, b9957d4086bc4a76b18ed9265a5739d5, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:18.123970 9720 raft_consensus.cc:2781] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:18.124087 9720 raft_consensus.cc:482] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:18.124184 9720 raft_consensus.cc:3037] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:18.126111 9720 raft_consensus.cc:504] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:18.126817 9720 leader_election.cc:288] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Requested vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:18.127388 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc87c8f97be84c05adad5e13ab5228b3" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:18.127393 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc87c8f97be84c05adad5e13ab5228b3" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:10:18.127609 4365 raft_consensus.cc:3037] T bc87c8f97be84c05adad5e13ab5228b3 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:18.127630 7420 raft_consensus.cc:3037] T bc87c8f97be84c05adad5e13ab5228b3 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:18.130285 4365 raft_consensus.cc:2454] T bc87c8f97be84c05adad5e13ab5228b3 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:18.130770 7420 raft_consensus.cc:2454] T bc87c8f97be84c05adad5e13ab5228b3 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:18.130859 7222 leader_election.cc:302] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:18.131335 9720 raft_consensus.cc:2781] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:18.131552 9720 raft_consensus.cc:686] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Becoming Leader. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Running, Role: LEADER
I20240501 20:10:18.131953 9720 consensus_queue.cc:237] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:18.135891 4092 catalog_manager.cc:5484] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71). New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:18.152119 4092 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:18.161993 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb1ca3dbf614f94b39e8ddd28ab04fc (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2cb1d6b4f2204c5c93a6b7b02e2c8a05]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:18.162389 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cb1ca3dbf614f94b39e8ddd28ab04fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:18.162351 4478 tablet_service.cc:1467] Processing CreateTablet for tablet e84a2462cb7e4e559fba887790a65a1f (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2cb1d6b4f2204c5c93a6b7b02e2c8a05]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:18.162786 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e84a2462cb7e4e559fba887790a65a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:18.167688 7267 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb1ca3dbf614f94b39e8ddd28ab04fc (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2cb1d6b4f2204c5c93a6b7b02e2c8a05]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:18.167670 7135 tablet_service.cc:1467] Processing CreateTablet for tablet 6cb1ca3dbf614f94b39e8ddd28ab04fc (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2cb1d6b4f2204c5c93a6b7b02e2c8a05]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:18.168279 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cb1ca3dbf614f94b39e8ddd28ab04fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:18.168298 7135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cb1ca3dbf614f94b39e8ddd28ab04fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:18.168879 9914 tablet_bootstrap.cc:492] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:18.170774 9914 tablet_bootstrap.cc:654] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:18.175515 7133 tablet_service.cc:1467] Processing CreateTablet for tablet e84a2462cb7e4e559fba887790a65a1f (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2cb1d6b4f2204c5c93a6b7b02e2c8a05]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:18.176018 7133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e84a2462cb7e4e559fba887790a65a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:18.179068 4215 tablet_service.cc:1467] Processing CreateTablet for tablet e84a2462cb7e4e559fba887790a65a1f (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2cb1d6b4f2204c5c93a6b7b02e2c8a05]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:18.179515 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e84a2462cb7e4e559fba887790a65a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:18.182427 7002 tablet_service.cc:1467] Processing CreateTablet for tablet 4a474e1f29bf4f059b22b2ec01f15579 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2cb1d6b4f2204c5c93a6b7b02e2c8a05]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:18.182749 9807 tablet_bootstrap.cc:492] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:18.182898 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a474e1f29bf4f059b22b2ec01f15579. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:18.183229 7134 tablet_service.cc:1467] Processing CreateTablet for tablet 4a474e1f29bf4f059b22b2ec01f15579 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2cb1d6b4f2204c5c93a6b7b02e2c8a05]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:18.183537 7134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a474e1f29bf4f059b22b2ec01f15579. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:18.184459 9807 tablet_bootstrap.cc:654] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:18.186349 9807 tablet_bootstrap.cc:492] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:18.186491 9807 ts_tablet_manager.cc:1367] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:10:18.187069 4609 tablet_service.cc:1467] Processing CreateTablet for tablet 4a474e1f29bf4f059b22b2ec01f15579 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=2cb1d6b4f2204c5c93a6b7b02e2c8a05]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:18.187506 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a474e1f29bf4f059b22b2ec01f15579. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:18.187337 9807 raft_consensus.cc:348] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:18.188197 9807 raft_consensus.cc:374] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:18.188295 9807 raft_consensus.cc:729] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:18.188551 9807 consensus_queue.cc:260] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:18.190425 9807 ts_tablet_manager.cc:1398] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:10:18.198051 9919 tablet_bootstrap.cc:492] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:18.200222 9919 tablet_bootstrap.cc:654] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:18.206534 9921 tablet_bootstrap.cc:492] T 4a474e1f29bf4f059b22b2ec01f15579 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:18.210440 9921 tablet_bootstrap.cc:654] T 4a474e1f29bf4f059b22b2ec01f15579 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:18.217128 9918 tablet_bootstrap.cc:492] T e84a2462cb7e4e559fba887790a65a1f P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:18.218530 9917 tablet_bootstrap.cc:492] T 4a474e1f29bf4f059b22b2ec01f15579 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:18.218724 9918 tablet_bootstrap.cc:654] T e84a2462cb7e4e559fba887790a65a1f P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:18.222513 9917 tablet_bootstrap.cc:654] T 4a474e1f29bf4f059b22b2ec01f15579 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:18.226864 9919 tablet_bootstrap.cc:492] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:18.227118 9919 ts_tablet_manager.cc:1367] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.005s
I20240501 20:10:18.228010 9918 tablet_bootstrap.cc:492] T e84a2462cb7e4e559fba887790a65a1f P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:10:18.228137 9918 ts_tablet_manager.cc:1367] T e84a2462cb7e4e559fba887790a65a1f P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20240501 20:10:18.228232 9914 tablet_bootstrap.cc:492] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:18.228403 9914 ts_tablet_manager.cc:1367] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.060s user 0.010s sys 0.016s
I20240501 20:10:18.229306 9914 raft_consensus.cc:348] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:18.229580 9918 raft_consensus.cc:348] T e84a2462cb7e4e559fba887790a65a1f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:18.229872 9918 raft_consensus.cc:374] T e84a2462cb7e4e559fba887790a65a1f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:18.229954 9918 raft_consensus.cc:729] T e84a2462cb7e4e559fba887790a65a1f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:10:18.230273 9918 consensus_queue.cc:260] T e84a2462cb7e4e559fba887790a65a1f P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:18.231380 9914 raft_consensus.cc:374] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:18.231493 9914 raft_consensus.cc:729] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:18.231978 9918 ts_tablet_manager.cc:1398] T e84a2462cb7e4e559fba887790a65a1f P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 20:10:18.231849 9914 consensus_queue.cc:260] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:18.235206 9914 ts_tablet_manager.cc:1398] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240501 20:10:18.235517 9914 tablet_bootstrap.cc:492] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:18.233037 9921 tablet_bootstrap.cc:492] T 4a474e1f29bf4f059b22b2ec01f15579 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:18.235898 9921 ts_tablet_manager.cc:1367] T 4a474e1f29bf4f059b22b2ec01f15579 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.003s
I20240501 20:10:18.231284 9201 raft_consensus.cc:1050] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd: attempting to promote NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b to VOTER
I20240501 20:10:18.238642 9914 tablet_bootstrap.cc:654] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:18.241156 9921 raft_consensus.cc:348] T 4a474e1f29bf4f059b22b2ec01f15579 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:18.237686 9201 consensus_queue.cc:237] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:18.246970 4498 raft_consensus.cc:1261] T 35c843bebb484f6cb655c146a02261ba P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:18.248245 9546 consensus_queue.cc:1035] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:18.236800 9919 raft_consensus.cc:348] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:18.251274 7420 raft_consensus.cc:1261] T 35c843bebb484f6cb655c146a02261ba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:18.261088 9919 raft_consensus.cc:374] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:18.251897 9546 consensus_queue.cc:1035] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:18.261798 9919 raft_consensus.cc:729] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:18.262321 9921 raft_consensus.cc:374] T 4a474e1f29bf4f059b22b2ec01f15579 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:18.262461 9921 raft_consensus.cc:729] T 4a474e1f29bf4f059b22b2ec01f15579 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:18.262778 9921 consensus_queue.cc:260] T 4a474e1f29bf4f059b22b2ec01f15579 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:18.263677 9919 consensus_queue.cc:260] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:18.264297 9919 ts_tablet_manager.cc:1398] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20240501 20:10:18.269403 9921 ts_tablet_manager.cc:1398] T 4a474e1f29bf4f059b22b2ec01f15579 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20240501 20:10:18.270210 9919 tablet_bootstrap.cc:492] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:18.272619 9919 tablet_bootstrap.cc:654] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:18.283154 9919 tablet_bootstrap.cc:492] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:18.283336 9919 ts_tablet_manager.cc:1367] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.002s
I20240501 20:10:18.284353 9919 raft_consensus.cc:348] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:18.284680 9919 raft_consensus.cc:374] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:18.284818 9919 raft_consensus.cc:729] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:18.285104 9919 consensus_queue.cc:260] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:18.285723 9919 ts_tablet_manager.cc:1398] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 20:10:18.288928 4092 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:18.290185 9917 tablet_bootstrap.cc:492] T 4a474e1f29bf4f059b22b2ec01f15579 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:18.290347 9917 ts_tablet_manager.cc:1367] T 4a474e1f29bf4f059b22b2ec01f15579 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.072s user 0.010s sys 0.010s
I20240501 20:10:18.291201 9917 raft_consensus.cc:348] T 4a474e1f29bf4f059b22b2ec01f15579 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:18.291405 9917 raft_consensus.cc:374] T 4a474e1f29bf4f059b22b2ec01f15579 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:18.291486 9917 raft_consensus.cc:729] T 4a474e1f29bf4f059b22b2ec01f15579 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:18.295794 9917 consensus_queue.cc:260] T 4a474e1f29bf4f059b22b2ec01f15579 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:18.298537 4365 raft_consensus.cc:1261] T bc87c8f97be84c05adad5e13ab5228b3 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:18.301753 9598 consensus_queue.cc:1035] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:18.304761 7420 raft_consensus.cc:1261] T bc87c8f97be84c05adad5e13ab5228b3 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:18.306658 9914 tablet_bootstrap.cc:492] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:18.306843 9914 ts_tablet_manager.cc:1367] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.071s user 0.005s sys 0.000s
I20240501 20:10:18.306926 9552 raft_consensus.cc:1050] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a: attempting to promote NON_VOTER db0742ee3860468590eefef6ae6074fd to VOTER
I20240501 20:10:18.307679 9552 consensus_queue.cc:237] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:18.307729 9914 raft_consensus.cc:348] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:18.307987 9914 raft_consensus.cc:374] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:18.308044 9914 raft_consensus.cc:729] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:18.308254 9914 consensus_queue.cc:260] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:18.308743 9914 ts_tablet_manager.cc:1398] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:18.310068 9919 tablet_bootstrap.cc:492] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:18.310388 9598 consensus_queue.cc:1035] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:18.314581 9919 tablet_bootstrap.cc:654] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:18.322233 4091 catalog_manager.cc:5576] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817) does not have the latest schema for tablet bc87c8f97be84c05adad5e13ab5228b3 (table rebalancer_dynamic_table [id=3696b5f0f4bb4fffa1e15ccf7bc4f8c8]). Expected version 1 got 0
I20240501 20:10:18.323685 9938 tablet.cc:1722] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:18.324505 9938 tablet.cc:1620] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:18.337568 4092 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=3696b5f0f4bb4fffa1e15ccf7bc4f8c8] alter complete (version 1)
W20240501 20:10:18.337255 7090 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.65:42941, user_credentials={real_user=slave}} blocked reactor thread for 72708.7us
I20240501 20:10:18.339093 7022 raft_consensus.cc:1261] T 35c843bebb484f6cb655c146a02261ba P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:18.339439 9939 tablet.cc:1722] T bc87c8f97be84c05adad5e13ab5228b3 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:18.339377 9926 raft_consensus.cc:2932] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:18.340600 9917 ts_tablet_manager.cc:1398] T 4a474e1f29bf4f059b22b2ec01f15579 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.050s user 0.003s sys 0.000s
I20240501 20:10:18.340837 9939 tablet.cc:1620] T bc87c8f97be84c05adad5e13ab5228b3 P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:18.354903 4087 catalog_manager.cc:5484] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:18.357390 9802 raft_consensus.cc:482] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:18.357564 9802 raft_consensus.cc:504] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:18.358336 9802 leader_election.cc:288] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:18.359509 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84a2462cb7e4e559fba887790a65a1f" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:18.359731 4498 raft_consensus.cc:2454] T e84a2462cb7e4e559fba887790a65a1f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:18.359943 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84a2462cb7e4e559fba887790a65a1f" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:18.360332 4169 leader_election.cc:302] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:18.360764 9802 raft_consensus.cc:2781] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:18.360850 9802 raft_consensus.cc:482] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:18.360918 9802 raft_consensus.cc:3037] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:10:18.361080 4167 leader_election.cc:341] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:18.363865 7416 raft_consensus.cc:2932] T 35c843bebb484f6cb655c146a02261ba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:18.371906 9201 consensus_queue.cc:1035] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:18.377084 7153 raft_consensus.cc:1261] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:18.377951 9552 consensus_queue.cc:1035] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:18.381275 4496 raft_consensus.cc:1261] T bb5700eb47e044c182c33183ebb27908 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:18.382504 4233 raft_consensus.cc:1261] T bb5700eb47e044c182c33183ebb27908 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:18.383713 9143 consensus_queue.cc:1035] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:18.384057 9744 consensus_queue.cc:1035] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:18.384972 4498 raft_consensus.cc:2932] T 35c843bebb484f6cb655c146a02261ba P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:18.390496 4091 catalog_manager.cc:5576] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689) does not have the latest schema for tablet bc87c8f97be84c05adad5e13ab5228b3 (table rebalancer_dynamic_table [id=3696b5f0f4bb4fffa1e15ccf7bc4f8c8]). Expected version 1 got 0
I20240501 20:10:18.393659 7021 raft_consensus.cc:2932] T 35c843bebb484f6cb655c146a02261ba P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:18.398828 9802 raft_consensus.cc:504] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:18.401335 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84a2462cb7e4e559fba887790a65a1f" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:10:18.401567 4499 raft_consensus.cc:3037] T e84a2462cb7e4e559fba887790a65a1f P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:18.404911 4499 raft_consensus.cc:2454] T e84a2462cb7e4e559fba887790a65a1f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:18.405558 4169 leader_election.cc:302] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:18.405988 9829 raft_consensus.cc:2781] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:18.406118 9829 raft_consensus.cc:686] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:10:18.406415 9829 consensus_queue.cc:237] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:18.408001 9950 tablet.cc:1722] T bc87c8f97be84c05adad5e13ab5228b3 P 10b47782558c41ee91f5e164a372049a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:18.408843 9950 tablet.cc:1620] T bc87c8f97be84c05adad5e13ab5228b3 P 10b47782558c41ee91f5e164a372049a: MemRowSet was empty: no flush needed.
I20240501 20:10:18.412807 4093 catalog_manager.cc:5576] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689) does not have the latest schema for tablet bc87c8f97be84c05adad5e13ab5228b3 (table rebalancer_dynamic_table [id=3696b5f0f4bb4fffa1e15ccf7bc4f8c8]). Expected version 1 got 0
I20240501 20:10:18.418661 9802 leader_election.cc:288] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:18.419184 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84a2462cb7e4e559fba887790a65a1f" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
W20240501 20:10:18.419675 4167 leader_election.cc:341] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:18.420671 9919 tablet_bootstrap.cc:492] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:18.420874 9919 ts_tablet_manager.cc:1367] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.111s user 0.019s sys 0.015s
I20240501 20:10:18.421943 9919 raft_consensus.cc:348] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:18.422339 9919 raft_consensus.cc:374] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:18.422474 9919 raft_consensus.cc:729] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:18.422799 9919 consensus_queue.cc:260] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:18.423606 9919 ts_tablet_manager.cc:1398] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:18.429319 4627 consensus_queue.cc:237] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:18.432730 4363 raft_consensus.cc:1261] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a [term 2 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:10:18.433023 7285 raft_consensus.cc:1261] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 20:10:18.430496 4091 catalog_manager.cc:5484] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:18.433668 9875 consensus_queue.cc:1035] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:18.435304 9875 consensus_queue.cc:1035] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:18.437395 9932 raft_consensus.cc:482] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:18.437556 9932 raft_consensus.cc:504] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:18.438409 9932 leader_election.cc:288] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:18.439229 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a474e1f29bf4f059b22b2ec01f15579" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:18.439493 4631 raft_consensus.cc:2454] T 4a474e1f29bf4f059b22b2ec01f15579 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:18.440090 7091 leader_election.cc:302] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:18.440424 7021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a474e1f29bf4f059b22b2ec01f15579" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:18.440676 7021 raft_consensus.cc:2454] T 4a474e1f29bf4f059b22b2ec01f15579 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:18.441155 9875 raft_consensus.cc:2932] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:18.446270 4088 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:18.448501 9932 raft_consensus.cc:2781] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:18.448662 9932 raft_consensus.cc:482] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:18.448760 9932 raft_consensus.cc:3037] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:18.448312 4363 raft_consensus.cc:2932] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:18.460359 4079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 65b0048212a745c08ab24ead346bf74e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:18.460582 7285 raft_consensus.cc:2932] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:18.466687 9938 tablet.cc:1722] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:18.467378 9938 tablet.cc:1620] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:18.471189 9932 raft_consensus.cc:504] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:18.472676 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a474e1f29bf4f059b22b2ec01f15579" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:10:18.472939 4627 raft_consensus.cc:3037] T 4a474e1f29bf4f059b22b2ec01f15579 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:18.477052 9552 raft_consensus.cc:2932] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:18.477979 4234 raft_consensus.cc:2932] T bb5700eb47e044c182c33183ebb27908 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:18.479403 9950 tablet.cc:1722] T bc87c8f97be84c05adad5e13ab5228b3 P 10b47782558c41ee91f5e164a372049a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:18.479959 9950 tablet.cc:1620] T bc87c8f97be84c05adad5e13ab5228b3 P 10b47782558c41ee91f5e164a372049a: MemRowSet was empty: no flush needed.
I20240501 20:10:18.485963 9932 leader_election.cc:288] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:18.480118 4627 raft_consensus.cc:2454] T 4a474e1f29bf4f059b22b2ec01f15579 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:18.486796 7021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a474e1f29bf4f059b22b2ec01f15579" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:18.487071 7021 raft_consensus.cc:3037] T 4a474e1f29bf4f059b22b2ec01f15579 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:18.487310 7091 leader_election.cc:302] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:18.487709 9932 raft_consensus.cc:2781] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:18.487910 9932 raft_consensus.cc:686] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Becoming Leader. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Running, Role: LEADER
I20240501 20:10:18.488292 9932 consensus_queue.cc:237] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:18.488480 4084 catalog_manager.cc:5484] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index 5 to 7, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New cstate: current_term: 2 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:18.489961 7021 raft_consensus.cc:2454] T 4a474e1f29bf4f059b22b2ec01f15579 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:18.491660 7153 consensus_queue.cc:237] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:18.491628 4498 raft_consensus.cc:2932] T bb5700eb47e044c182c33183ebb27908 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:18.494024 9939 tablet.cc:1722] T bc87c8f97be84c05adad5e13ab5228b3 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:18.494535 9939 tablet.cc:1620] T bc87c8f97be84c05adad5e13ab5228b3 P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:18.494736 7153 raft_consensus.cc:2932] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:18.495400 7416 raft_consensus.cc:1261] T 35c843bebb484f6cb655c146a02261ba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 20:10:18.499449 4167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.70:36321, user_credentials={real_user=slave}} blocked reactor thread for 63364.7us
I20240501 20:10:18.501049 9201 consensus_queue.cc:1035] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:18.503720 4093 catalog_manager.cc:5484] T bb5700eb47e044c182c33183ebb27908 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 4 to 5, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:18.505152 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet 65b0048212a745c08ab24ead346bf74e with delete_type TABLET_DATA_TOMBSTONED (TS 3e974c03d3c546f9ab6c91e762a5426b not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:18.507872 9201 raft_consensus.cc:2932] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:18.513239 4084 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=3696b5f0f4bb4fffa1e15ccf7bc4f8c8] alter complete (version 2)
I20240501 20:10:18.515008 7419 raft_consensus.cc:2932] T 35c843bebb484f6cb655c146a02261ba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:18.515525 4498 raft_consensus.cc:1261] T 35c843bebb484f6cb655c146a02261ba P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:18.518345 9958 tablet_replica.cc:330] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:18.518718 9958 raft_consensus.cc:2229] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:18.519011 9958 raft_consensus.cc:2258] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:18.519040 4365 consensus_queue.cc:237] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:18.520218 9958 ts_tablet_manager.cc:1875] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:18.530439 9958 ts_tablet_manager.cc:1888] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240501 20:10:18.530719 9958 log.cc:1196] T 65b0048212a745c08ab24ead346bf74e P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/65b0048212a745c08ab24ead346bf74e
I20240501 20:10:18.530825 4092 catalog_manager.cc:5484] T 35c843bebb484f6cb655c146a02261ba P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:18.531819 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet 65b0048212a745c08ab24ead346bf74e (table rebalance_test_table_9 [id=eef6790ef9874038b63d81f758f188d7]) successfully deleted
I20240501 20:10:18.531958 4076 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 35c843bebb484f6cb655c146a02261ba with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:18.532524 9201 consensus_queue.cc:1035] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:18.534492 4498 raft_consensus.cc:2932] T 35c843bebb484f6cb655c146a02261ba P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:18.535758 7002 tablet_service.cc:1514] Processing DeleteTablet for tablet 35c843bebb484f6cb655c146a02261ba with delete_type TABLET_DATA_TOMBSTONED (TS ef0a4ada548446b3a531cb39a3c4e38d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:18.534801 7153 raft_consensus.cc:1261] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:18.537178 9552 consensus_queue.cc:1035] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:18.537781 4233 raft_consensus.cc:1261] T bb5700eb47e044c182c33183ebb27908 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:18.540109 9552 consensus_queue.cc:1035] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:18.543936 4234 raft_consensus.cc:2932] T bb5700eb47e044c182c33183ebb27908 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:18.554636 4087 catalog_manager.cc:5484] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd reported cstate change: term changed from 0 to 1, leader changed from <none> to db0742ee3860468590eefef6ae6074fd (127.29.219.70). New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:18.558831 7155 raft_consensus.cc:2932] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:18.561313 9932 raft_consensus.cc:482] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:18.561558 9932 raft_consensus.cc:504] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:18.567790 4087 catalog_manager.cc:5484] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:18.569379 4496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84a2462cb7e4e559fba887790a65a1f" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:18.569640 4496 raft_consensus.cc:2379] T e84a2462cb7e4e559fba887790a65a1f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db0742ee3860468590eefef6ae6074fd in current term 1: Already voted for candidate dca57823003e4729b769ca12367d7530 in this term.
I20240501 20:10:18.572311 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet bb5700eb47e044c182c33183ebb27908 with delete_type TABLET_DATA_TOMBSTONED (TS 3e974c03d3c546f9ab6c91e762a5426b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:18.572788 9958 tablet_replica.cc:330] T bb5700eb47e044c182c33183ebb27908 P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:18.573012 9958 raft_consensus.cc:2229] T bb5700eb47e044c182c33183ebb27908 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:18.573344 9958 raft_consensus.cc:2258] T bb5700eb47e044c182c33183ebb27908 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:18.543287 9143 raft_consensus.cc:2932] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:18.573925 9959 tablet_replica.cc:330] T 35c843bebb484f6cb655c146a02261ba P ef0a4ada548446b3a531cb39a3c4e38d: stopping tablet replica
I20240501 20:10:18.574218 9959 raft_consensus.cc:2229] T 35c843bebb484f6cb655c146a02261ba P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:18.574473 9959 raft_consensus.cc:2258] T 35c843bebb484f6cb655c146a02261ba P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:18.574595 9958 ts_tablet_manager.cc:1875] T bb5700eb47e044c182c33183ebb27908 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:18.575639 9959 ts_tablet_manager.cc:1875] T 35c843bebb484f6cb655c146a02261ba P ef0a4ada548446b3a531cb39a3c4e38d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:18.581033 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bb5700eb47e044c182c33183ebb27908 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:18.589478 9959 ts_tablet_manager.cc:1888] T 35c843bebb484f6cb655c146a02261ba P ef0a4ada548446b3a531cb39a3c4e38d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:18.589953 9959 log.cc:1196] T 35c843bebb484f6cb655c146a02261ba P ef0a4ada548446b3a531cb39a3c4e38d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/wals/35c843bebb484f6cb655c146a02261ba
I20240501 20:10:18.590703 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84a2462cb7e4e559fba887790a65a1f" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:18.591516 7090 leader_election.cc:302] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db0742ee3860468590eefef6ae6074fd; no voters: 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530
I20240501 20:10:18.593883 9929 raft_consensus.cc:3037] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:18.596819 9929 raft_consensus.cc:2726] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:18.601179 9958 ts_tablet_manager.cc:1888] T bb5700eb47e044c182c33183ebb27908 P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 20:10:18.601467 9932 leader_election.cc:288] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:18.610760 9958 log.cc:1196] T bb5700eb47e044c182c33183ebb27908 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/bb5700eb47e044c182c33183ebb27908
I20240501 20:10:18.616712 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet bb5700eb47e044c182c33183ebb27908 (table rebalance_test_table_17 [id=09dd73eba68c4f8499a786750d39c1cf]) successfully deleted
I20240501 20:10:18.617141 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet 35c843bebb484f6cb655c146a02261ba (table rebalancer_extra_table_6 [id=f56b02450c6f407b827d0131f37c9ffa]) successfully deleted
I20240501 20:10:18.621847 4091 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:18.632967 9938 tablet.cc:1722] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:18.639986 9939 tablet.cc:1722] T bc87c8f97be84c05adad5e13ab5228b3 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:18.643797 8785 raft_consensus.cc:482] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:18.643970 8785 raft_consensus.cc:504] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:18.644817 8785 leader_election.cc:288] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:18.646730 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb1ca3dbf614f94b39e8ddd28ab04fc" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:18.646981 7285 raft_consensus.cc:2454] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:18.647461 4168 leader_election.cc:302] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:18.648584 9950 tablet.cc:1722] T bc87c8f97be84c05adad5e13ab5228b3 P 10b47782558c41ee91f5e164a372049a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:18.648692 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb1ca3dbf614f94b39e8ddd28ab04fc" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:18.648949 7155 raft_consensus.cc:2454] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:18.654789 8785 raft_consensus.cc:2781] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:18.654935 8785 raft_consensus.cc:482] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:18.655004 8785 raft_consensus.cc:3037] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:18.657725 8785 raft_consensus.cc:504] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:18.660581 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb1ca3dbf614f94b39e8ddd28ab04fc" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:18.660805 7285 raft_consensus.cc:3037] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:18.661175 4092 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table_new_name [id=3696b5f0f4bb4fffa1e15ccf7bc4f8c8] alter complete (version 3)
I20240501 20:10:18.663753 7285 raft_consensus.cc:2454] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:18.664472 4168 leader_election.cc:302] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:18.664767 9953 raft_consensus.cc:2781] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:18.664871 9953 raft_consensus.cc:686] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:10:18.665164 9953 consensus_queue.cc:237] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:18.671368 8785 leader_election.cc:288] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:18.673619 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cb1ca3dbf614f94b39e8ddd28ab04fc" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:18.673961 7155 raft_consensus.cc:3037] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:18.682529 4091 catalog_manager.cc:5484] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:18.689993 7155 raft_consensus.cc:2454] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:18.766464 4091 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:18.766681 4091 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:18.769428 4091 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet bc87c8f97be84c05adad5e13ab5228b3
I20240501 20:10:18.770330 4346 tablet_service.cc:1514] Processing DeleteTablet for tablet bc87c8f97be84c05adad5e13ab5228b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:18 UTC) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:18.770887 9983 tablet_replica.cc:330] T bc87c8f97be84c05adad5e13ab5228b3 P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:18.771092 7400 tablet_service.cc:1514] Processing DeleteTablet for tablet bc87c8f97be84c05adad5e13ab5228b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:18 UTC) from {username='slave'} at 127.0.0.1:40182
I20240501 20:10:18.771173 9983 raft_consensus.cc:2229] T bc87c8f97be84c05adad5e13ab5228b3 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:18.771791 7267 tablet_service.cc:1514] Processing DeleteTablet for tablet bc87c8f97be84c05adad5e13ab5228b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:18 UTC) from {username='slave'} at 127.0.0.1:54166
I20240501 20:10:18.772585 9983 raft_consensus.cc:2258] T bc87c8f97be84c05adad5e13ab5228b3 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:18.772723 9985 tablet_replica.cc:330] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:18.772966 9985 raft_consensus.cc:2229] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:18.773423 9985 raft_consensus.cc:2258] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:18.773550 9983 ts_tablet_manager.cc:1875] T bc87c8f97be84c05adad5e13ab5228b3 P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:18.777143 9985 ts_tablet_manager.cc:1875] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:18.781186 9984 tablet_replica.cc:330] T bc87c8f97be84c05adad5e13ab5228b3 P b9957d4086bc4a76b18ed9265a5739d5: stopping tablet replica
I20240501 20:10:18.781399 9984 raft_consensus.cc:2229] T bc87c8f97be84c05adad5e13ab5228b3 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:18.782158 9984 raft_consensus.cc:2258] T bc87c8f97be84c05adad5e13ab5228b3 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:18.785984 9953 consensus_queue.cc:1035] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:18.787016 9984 ts_tablet_manager.cc:1875] T bc87c8f97be84c05adad5e13ab5228b3 P b9957d4086bc4a76b18ed9265a5739d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:18.791724 9983 ts_tablet_manager.cc:1888] T bc87c8f97be84c05adad5e13ab5228b3 P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:18.793386 9983 log.cc:1196] T bc87c8f97be84c05adad5e13ab5228b3 P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/bc87c8f97be84c05adad5e13ab5228b3
I20240501 20:10:18.793653 9984 ts_tablet_manager.cc:1888] T bc87c8f97be84c05adad5e13ab5228b3 P b9957d4086bc4a76b18ed9265a5739d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:18.793812 9984 log.cc:1196] T bc87c8f97be84c05adad5e13ab5228b3 P b9957d4086bc4a76b18ed9265a5739d5: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/wals/bc87c8f97be84c05adad5e13ab5228b3
I20240501 20:10:18.794032 9983 ts_tablet_manager.cc:1909] T bc87c8f97be84c05adad5e13ab5228b3 P 10b47782558c41ee91f5e164a372049a: Deleting consensus metadata
I20240501 20:10:18.794281 9984 ts_tablet_manager.cc:1909] T bc87c8f97be84c05adad5e13ab5228b3 P b9957d4086bc4a76b18ed9265a5739d5: Deleting consensus metadata
I20240501 20:10:18.795307 4079 catalog_manager.cc:4830] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): tablet bc87c8f97be84c05adad5e13ab5228b3 (table rebalancer_dynamic_table_new_name [id=3696b5f0f4bb4fffa1e15ccf7bc4f8c8]) successfully deleted
I20240501 20:10:18.795284 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet bc87c8f97be84c05adad5e13ab5228b3 (table rebalancer_dynamic_table_new_name [id=3696b5f0f4bb4fffa1e15ccf7bc4f8c8]) successfully deleted
I20240501 20:10:18.795883 9985 ts_tablet_manager.cc:1888] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:18.796001 9985 log.cc:1196] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/bc87c8f97be84c05adad5e13ab5228b3
I20240501 20:10:18.796401 9985 ts_tablet_manager.cc:1909] T bc87c8f97be84c05adad5e13ab5228b3 P bceeae1c79964acabdae8ff771bf0c97: Deleting consensus metadata
I20240501 20:10:18.797287 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet bc87c8f97be84c05adad5e13ab5228b3 (table rebalancer_dynamic_table_new_name [id=3696b5f0f4bb4fffa1e15ccf7bc4f8c8]) successfully deleted
I20240501 20:10:18.813249 8785 consensus_queue.cc:1035] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:18.877954 4087 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:18.886912 4478 tablet_service.cc:1467] Processing CreateTablet for tablet 58afedbe61974f4c81e80b4648c18f5e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9b6544bc81b445938dcf75f7ae09cd87]), partition=
I20240501 20:10:18.887295 7267 tablet_service.cc:1467] Processing CreateTablet for tablet 58afedbe61974f4c81e80b4648c18f5e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9b6544bc81b445938dcf75f7ae09cd87]), partition=
I20240501 20:10:18.887338 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58afedbe61974f4c81e80b4648c18f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:18.887672 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58afedbe61974f4c81e80b4648c18f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:18.892367 7002 tablet_service.cc:1467] Processing CreateTablet for tablet 58afedbe61974f4c81e80b4648c18f5e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9b6544bc81b445938dcf75f7ae09cd87]), partition=
I20240501 20:10:18.892781 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58afedbe61974f4c81e80b4648c18f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:18.907598 9932 consensus_queue.cc:1035] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:18.915463 10016 tablet_bootstrap.cc:492] T 58afedbe61974f4c81e80b4648c18f5e P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:18.917869 10016 tablet_bootstrap.cc:654] T 58afedbe61974f4c81e80b4648c18f5e P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:18.922258 10013 tablet_bootstrap.cc:492] T 58afedbe61974f4c81e80b4648c18f5e P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:18.923880 10013 tablet_bootstrap.cc:654] T 58afedbe61974f4c81e80b4648c18f5e P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:18.943612 9926 consensus_queue.cc:1035] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:18.952641 10013 tablet_bootstrap.cc:492] T 58afedbe61974f4c81e80b4648c18f5e P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:10:18.952914 10013 ts_tablet_manager.cc:1367] T 58afedbe61974f4c81e80b4648c18f5e P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.002s
I20240501 20:10:18.949872 10016 tablet_bootstrap.cc:492] T 58afedbe61974f4c81e80b4648c18f5e P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:18.954110 10016 ts_tablet_manager.cc:1367] T 58afedbe61974f4c81e80b4648c18f5e P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.000s
I20240501 20:10:18.953879 10013 raft_consensus.cc:348] T 58afedbe61974f4c81e80b4648c18f5e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:18.954344 10013 raft_consensus.cc:374] T 58afedbe61974f4c81e80b4648c18f5e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:18.954469 10013 raft_consensus.cc:729] T 58afedbe61974f4c81e80b4648c18f5e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:10:18.954797 10013 consensus_queue.cc:260] T 58afedbe61974f4c81e80b4648c18f5e P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:18.955673 10013 ts_tablet_manager.cc:1398] T 58afedbe61974f4c81e80b4648c18f5e P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:18.956943 10016 raft_consensus.cc:348] T 58afedbe61974f4c81e80b4648c18f5e P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:18.957630 10016 raft_consensus.cc:374] T 58afedbe61974f4c81e80b4648c18f5e P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:18.957950 10016 raft_consensus.cc:729] T 58afedbe61974f4c81e80b4648c18f5e P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:18.958380 10016 consensus_queue.cc:260] T 58afedbe61974f4c81e80b4648c18f5e P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:18.965272 10016 ts_tablet_manager.cc:1398] T 58afedbe61974f4c81e80b4648c18f5e P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 20:10:18.977310 10014 tablet_bootstrap.cc:492] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:18.999603 10014 tablet_bootstrap.cc:654] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:19.026515 10014 tablet_bootstrap.cc:492] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:19.026837 10014 ts_tablet_manager.cc:1367] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.050s user 0.006s sys 0.000s
I20240501 20:10:19.028385 10014 raft_consensus.cc:348] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:19.028744 10014 raft_consensus.cc:374] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:19.028916 10014 raft_consensus.cc:729] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:19.029306 10014 consensus_queue.cc:260] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:19.035514 10014 ts_tablet_manager.cc:1398] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 20:10:19.055430 8785 consensus_queue.cc:1035] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:19.061378 9802 consensus_queue.cc:1035] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:19.110765 4346 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:19.114954 4609 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 20:10:19.126986 9800 raft_consensus.cc:482] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:19.127239 9800 raft_consensus.cc:504] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:19.128304 9800 leader_election.cc:288] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:19.128816 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58afedbe61974f4c81e80b4648c18f5e" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:19.129067 7285 raft_consensus.cc:2454] T 58afedbe61974f4c81e80b4648c18f5e P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:19.129662 4496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58afedbe61974f4c81e80b4648c18f5e" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:19.129940 4496 raft_consensus.cc:2454] T 58afedbe61974f4c81e80b4648c18f5e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:19.129983 6956 leader_election.cc:302] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:19.130493 9800 raft_consensus.cc:2781] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:19.130628 9800 raft_consensus.cc:482] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:19.130728 9800 raft_consensus.cc:3037] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:19.135538 9800 raft_consensus.cc:504] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:19.136557 4496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58afedbe61974f4c81e80b4648c18f5e" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:10:19.136756 4496 raft_consensus.cc:3037] T 58afedbe61974f4c81e80b4648c18f5e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:19.140916 9800 leader_election.cc:288] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:19.144135 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58afedbe61974f4c81e80b4648c18f5e" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:19.144374 7285 raft_consensus.cc:3037] T 58afedbe61974f4c81e80b4648c18f5e P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:19.151234 7285 raft_consensus.cc:2454] T 58afedbe61974f4c81e80b4648c18f5e P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:19.151611 4496 raft_consensus.cc:2454] T 58afedbe61974f4c81e80b4648c18f5e P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:19.152458 6956 leader_election.cc:302] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:19.153272 9943 raft_consensus.cc:2781] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:19.153437 9943 raft_consensus.cc:686] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Becoming Leader. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Running, Role: LEADER
I20240501 20:10:19.153738 9943 consensus_queue.cc:237] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:19.178407 4092 catalog_manager.cc:5484] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69). New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:19.189642 4478 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 20:10:19.192842 7355 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.65:42941, user_credentials={real_user=slave}} blocked reactor thread for 52448.2us
I20240501 20:10:19.193608 7400 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:10:19.252030 7133 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:19.273952 4092 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:19.290911 4092 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:19.292124 7267 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:10:19.308413 7287 raft_consensus.cc:1261] T 58afedbe61974f4c81e80b4648c18f5e P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:19.309986 4496 raft_consensus.cc:1261] T 58afedbe61974f4c81e80b4648c18f5e P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:19.312320 9800 consensus_queue.cc:1035] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:19.313508 9800 consensus_queue.cc:1035] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:19.317010 7400 tablet_service.cc:1467] Processing CreateTablet for tablet 09dcd419921941e9a19cfd9a20b89c70 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=ccbf7dce50cd4324a764233997f8c643]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:19.317512 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09dcd419921941e9a19cfd9a20b89c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:19.318457 4478 tablet_service.cc:1467] Processing CreateTablet for tablet 09dcd419921941e9a19cfd9a20b89c70 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=ccbf7dce50cd4324a764233997f8c643]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:19.318912 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09dcd419921941e9a19cfd9a20b89c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:19.323968 4477 tablet_service.cc:1467] Processing CreateTablet for tablet ce0b97a1badb4935a4553902e6feb20b (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=ccbf7dce50cd4324a764233997f8c643]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:19.324677 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce0b97a1badb4935a4553902e6feb20b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:19.324594 4215 tablet_service.cc:1467] Processing CreateTablet for tablet c9025370d3f44514a1430375e27c0239 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=ccbf7dce50cd4324a764233997f8c643]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:19.324992 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9025370d3f44514a1430375e27c0239. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:19.321575 4346 tablet_service.cc:1467] Processing CreateTablet for tablet c9025370d3f44514a1430375e27c0239 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=ccbf7dce50cd4324a764233997f8c643]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:19.327342 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9025370d3f44514a1430375e27c0239. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:19.327560 4347 tablet_service.cc:1467] Processing CreateTablet for tablet ce0b97a1badb4935a4553902e6feb20b (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=ccbf7dce50cd4324a764233997f8c643]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:19.327950 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce0b97a1badb4935a4553902e6feb20b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:19.332195 7267 tablet_service.cc:1467] Processing CreateTablet for tablet ce0b97a1badb4935a4553902e6feb20b (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=ccbf7dce50cd4324a764233997f8c643]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:19.332895 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce0b97a1badb4935a4553902e6feb20b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:19.339027 7001 tablet_service.cc:1467] Processing CreateTablet for tablet c9025370d3f44514a1430375e27c0239 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=ccbf7dce50cd4324a764233997f8c643]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:19.339499 7001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9025370d3f44514a1430375e27c0239. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:19.343561 7002 tablet_service.cc:1467] Processing CreateTablet for tablet 09dcd419921941e9a19cfd9a20b89c70 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=ccbf7dce50cd4324a764233997f8c643]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:19.343952 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09dcd419921941e9a19cfd9a20b89c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:19.393173 10016 tablet_bootstrap.cc:492] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:19.395068 10016 tablet_bootstrap.cc:654] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:19.398245 10013 tablet_bootstrap.cc:492] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:19.399613 10013 tablet_bootstrap.cc:654] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:19.410249 10058 tablet_bootstrap.cc:492] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:19.416776 4087 catalog_manager.cc:5576] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447) does not have the latest schema for tablet 58afedbe61974f4c81e80b4648c18f5e (table rebalancer_dynamic_table [id=9b6544bc81b445938dcf75f7ae09cd87]). Expected version 1 got 0
I20240501 20:10:19.419302 10013 tablet_bootstrap.cc:492] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:10:19.419559 10014 tablet_bootstrap.cc:492] T c9025370d3f44514a1430375e27c0239 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:19.421344 10014 tablet_bootstrap.cc:654] T c9025370d3f44514a1430375e27c0239 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:19.427997 10056 tablet_bootstrap.cc:492] T ce0b97a1badb4935a4553902e6feb20b P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:19.426754 4215 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:19.435382 10058 tablet_bootstrap.cc:654] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:19.435367 10013 ts_tablet_manager.cc:1367] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.003s
I20240501 20:10:19.436082 10013 raft_consensus.cc:348] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:19.436323 10013 raft_consensus.cc:374] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:19.436401 10013 raft_consensus.cc:729] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:10:19.436641 10013 consensus_queue.cc:260] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:19.437261 10013 ts_tablet_manager.cc:1398] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:19.437557 10013 tablet_bootstrap.cc:492] T ce0b97a1badb4935a4553902e6feb20b P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:19.442909 4087 catalog_manager.cc:5576] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595) does not have the latest schema for tablet 58afedbe61974f4c81e80b4648c18f5e (table rebalancer_dynamic_table [id=9b6544bc81b445938dcf75f7ae09cd87]). Expected version 1 got 0
I20240501 20:10:19.456914 10054 tablet_bootstrap.cc:492] T 09dcd419921941e9a19cfd9a20b89c70 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:19.460795 10056 tablet_bootstrap.cc:654] T ce0b97a1badb4935a4553902e6feb20b P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:19.461977 10013 tablet_bootstrap.cc:654] T ce0b97a1badb4935a4553902e6feb20b P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:19.469048 10016 tablet_bootstrap.cc:492] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:19.469242 10016 ts_tablet_manager.cc:1367] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.076s user 0.004s sys 0.005s
I20240501 20:10:19.472558 10016 raft_consensus.cc:348] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:19.472817 10016 raft_consensus.cc:374] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:19.472944 10016 raft_consensus.cc:729] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:19.473196 10016 consensus_queue.cc:260] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:19.473838 10016 ts_tablet_manager.cc:1398] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240501 20:10:19.477449 10066 tablet.cc:1722] T 58afedbe61974f4c81e80b4648c18f5e P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:19.480983 10014 tablet_bootstrap.cc:492] T c9025370d3f44514a1430375e27c0239 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:19.481159 10014 ts_tablet_manager.cc:1367] T c9025370d3f44514a1430375e27c0239 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.062s user 0.009s sys 0.000s
I20240501 20:10:19.481557 4089 catalog_manager.cc:5576] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447) does not have the latest schema for tablet 58afedbe61974f4c81e80b4648c18f5e (table rebalancer_dynamic_table [id=9b6544bc81b445938dcf75f7ae09cd87]). Expected version 1 got 0
I20240501 20:10:19.482650 10064 tablet.cc:1722] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:19.483321 10064 tablet.cc:1620] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:19.483424 10054 tablet_bootstrap.cc:654] T 09dcd419921941e9a19cfd9a20b89c70 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:19.486992 4087 catalog_manager.cc:5576] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237) does not have the latest schema for tablet 58afedbe61974f4c81e80b4648c18f5e (table rebalancer_dynamic_table [id=9b6544bc81b445938dcf75f7ae09cd87]). Expected version 1 got 0
I20240501 20:10:19.489053 10014 raft_consensus.cc:348] T c9025370d3f44514a1430375e27c0239 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:19.489303 10014 raft_consensus.cc:374] T c9025370d3f44514a1430375e27c0239 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:19.489378 10014 raft_consensus.cc:729] T c9025370d3f44514a1430375e27c0239 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:19.489629 10014 consensus_queue.cc:260] T c9025370d3f44514a1430375e27c0239 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:19.491891 10014 ts_tablet_manager.cc:1398] T c9025370d3f44514a1430375e27c0239 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240501 20:10:19.492609 10014 tablet_bootstrap.cc:492] T 09dcd419921941e9a19cfd9a20b89c70 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:19.505532 10058 tablet_bootstrap.cc:492] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:19.509443 10066 tablet.cc:1620] T 58afedbe61974f4c81e80b4648c18f5e P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:19.510597 10058 ts_tablet_manager.cc:1367] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.100s user 0.003s sys 0.003s
I20240501 20:10:19.510803 10067 tablet.cc:1722] T 58afedbe61974f4c81e80b4648c18f5e P 3e974c03d3c546f9ab6c91e762a5426b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:19.512907 4087 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=9b6544bc81b445938dcf75f7ae09cd87] alter complete (version 1)
I20240501 20:10:19.514745 10054 tablet_bootstrap.cc:492] T 09dcd419921941e9a19cfd9a20b89c70 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:19.517115 7002 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:19.518389 10054 ts_tablet_manager.cc:1367] T 09dcd419921941e9a19cfd9a20b89c70 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.062s user 0.006s sys 0.000s
I20240501 20:10:19.518779 10058 raft_consensus.cc:348] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:19.519095 10067 tablet.cc:1620] T 58afedbe61974f4c81e80b4648c18f5e P 3e974c03d3c546f9ab6c91e762a5426b: MemRowSet was empty: no flush needed.
I20240501 20:10:19.519917 10054 raft_consensus.cc:348] T 09dcd419921941e9a19cfd9a20b89c70 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:19.520210 10058 raft_consensus.cc:374] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:19.520359 10058 raft_consensus.cc:729] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:19.520690 10058 consensus_queue.cc:260] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:19.521358 10013 tablet_bootstrap.cc:492] T ce0b97a1badb4935a4553902e6feb20b P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:10:19.521363 10058 ts_tablet_manager.cc:1398] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240501 20:10:19.521477 10054 raft_consensus.cc:374] T 09dcd419921941e9a19cfd9a20b89c70 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:19.521747 10054 raft_consensus.cc:729] T 09dcd419921941e9a19cfd9a20b89c70 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:19.522698 10014 tablet_bootstrap.cc:654] T 09dcd419921941e9a19cfd9a20b89c70 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:19.524330 10013 ts_tablet_manager.cc:1367] T ce0b97a1badb4935a4553902e6feb20b P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.087s user 0.007s sys 0.004s
I20240501 20:10:19.524686 10054 consensus_queue.cc:260] T 09dcd419921941e9a19cfd9a20b89c70 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:19.525027 10013 raft_consensus.cc:348] T ce0b97a1badb4935a4553902e6feb20b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:19.525246 10013 raft_consensus.cc:374] T ce0b97a1badb4935a4553902e6feb20b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:19.525321 10013 raft_consensus.cc:729] T ce0b97a1badb4935a4553902e6feb20b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:10:19.535207 10013 consensus_queue.cc:260] T ce0b97a1badb4935a4553902e6feb20b P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:19.537029 10013 ts_tablet_manager.cc:1398] T ce0b97a1badb4935a4553902e6feb20b P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240501 20:10:19.539829 10014 tablet_bootstrap.cc:492] T 09dcd419921941e9a19cfd9a20b89c70 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:19.539999 10014 ts_tablet_manager.cc:1367] T 09dcd419921941e9a19cfd9a20b89c70 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.000s
I20240501 20:10:19.540479 4089 catalog_manager.cc:5576] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595) does not have the latest schema for tablet 58afedbe61974f4c81e80b4648c18f5e (table rebalancer_dynamic_table [id=9b6544bc81b445938dcf75f7ae09cd87]). Expected version 1 got 0
I20240501 20:10:19.543988 10014 raft_consensus.cc:348] T 09dcd419921941e9a19cfd9a20b89c70 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:19.544216 10014 raft_consensus.cc:374] T 09dcd419921941e9a19cfd9a20b89c70 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:19.544323 10014 raft_consensus.cc:729] T 09dcd419921941e9a19cfd9a20b89c70 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:19.544564 10014 consensus_queue.cc:260] T 09dcd419921941e9a19cfd9a20b89c70 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:19.549342 10014 ts_tablet_manager.cc:1398] T 09dcd419921941e9a19cfd9a20b89c70 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 20:10:19.555822 10054 ts_tablet_manager.cc:1398] T 09dcd419921941e9a19cfd9a20b89c70 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.037s user 0.002s sys 0.000s
I20240501 20:10:19.556991 10056 tablet_bootstrap.cc:492] T ce0b97a1badb4935a4553902e6feb20b P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:10:19.557533 10056 ts_tablet_manager.cc:1367] T ce0b97a1badb4935a4553902e6feb20b P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.130s user 0.002s sys 0.004s
I20240501 20:10:19.559213 10056 raft_consensus.cc:348] T ce0b97a1badb4935a4553902e6feb20b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:19.559427 10056 raft_consensus.cc:374] T ce0b97a1badb4935a4553902e6feb20b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:19.559518 10056 raft_consensus.cc:729] T ce0b97a1badb4935a4553902e6feb20b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:10:19.559808 10056 consensus_queue.cc:260] T ce0b97a1badb4935a4553902e6feb20b P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:19.560364 10056 ts_tablet_manager.cc:1398] T ce0b97a1badb4935a4553902e6feb20b P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:10:19.560878 10056 tablet_bootstrap.cc:492] T c9025370d3f44514a1430375e27c0239 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:19.562741 10056 tablet_bootstrap.cc:654] T c9025370d3f44514a1430375e27c0239 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:19.563653 9940 raft_consensus.cc:482] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:19.563814 9940 raft_consensus.cc:504] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:19.564607 9940 leader_election.cc:288] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:19.565968 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09dcd419921941e9a19cfd9a20b89c70" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:19.566237 7022 raft_consensus.cc:2454] T 09dcd419921941e9a19cfd9a20b89c70 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:10:19.566572 7416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09dcd419921941e9a19cfd9a20b89c70" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:19.566831 7416 raft_consensus.cc:2454] T 09dcd419921941e9a19cfd9a20b89c70 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:10:19.567245 4434 leader_election.cc:302] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:19.567792 9319 raft_consensus.cc:2781] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:19.567891 9319 raft_consensus.cc:482] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:19.567973 9319 raft_consensus.cc:3037] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:19.572153 9319 raft_consensus.cc:504] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:19.573055 7416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09dcd419921941e9a19cfd9a20b89c70" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:19.573259 7416 raft_consensus.cc:3037] T 09dcd419921941e9a19cfd9a20b89c70 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:19.574668 9956 raft_consensus.cc:482] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:19.574841 9956 raft_consensus.cc:504] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:19.575547 9956 leader_election.cc:288] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:19.576295 4363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0b97a1badb4935a4553902e6feb20b" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:10:19.576465 4496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0b97a1badb4935a4553902e6feb20b" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:19.576529 4363 raft_consensus.cc:2454] T ce0b97a1badb4935a4553902e6feb20b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:19.576664 4496 raft_consensus.cc:2454] T ce0b97a1badb4935a4553902e6feb20b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:19.577140 7222 leader_election.cc:302] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:19.577728 9956 raft_consensus.cc:2781] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:19.577853 9956 raft_consensus.cc:482] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:19.577930 9956 raft_consensus.cc:3037] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:19.580680 9956 raft_consensus.cc:504] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:19.583624 4363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0b97a1badb4935a4553902e6feb20b" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:10:19.583858 4363 raft_consensus.cc:3037] T ce0b97a1badb4935a4553902e6feb20b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:19.585063 9319 leader_election.cc:288] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Requested vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:19.585572 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09dcd419921941e9a19cfd9a20b89c70" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:19.585794 7022 raft_consensus.cc:3037] T 09dcd419921941e9a19cfd9a20b89c70 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:19.585683 7416 raft_consensus.cc:2454] T 09dcd419921941e9a19cfd9a20b89c70 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:10:19.586400 4435 leader_election.cc:302] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, b9957d4086bc4a76b18ed9265a5739d5; no voters:
I20240501 20:10:19.586794 9319 raft_consensus.cc:2781] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:19.586980 9319 raft_consensus.cc:686] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:10:19.587424 9319 consensus_queue.cc:237] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:19.588745 7022 raft_consensus.cc:2454] T 09dcd419921941e9a19cfd9a20b89c70 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:10:19.591037 4496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce0b97a1badb4935a4553902e6feb20b" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:10:19.591269 4496 raft_consensus.cc:3037] T ce0b97a1badb4935a4553902e6feb20b P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:19.586546 4363 raft_consensus.cc:2454] T ce0b97a1badb4935a4553902e6feb20b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:19.593389 7222 leader_election.cc:302] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:19.594178 4496 raft_consensus.cc:2454] T ce0b97a1badb4935a4553902e6feb20b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:19.595224 9956 leader_election.cc:288] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:19.595443 9956 raft_consensus.cc:2781] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:19.595558 9956 raft_consensus.cc:686] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Becoming Leader. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Running, Role: LEADER
I20240501 20:10:19.595826 9956 consensus_queue.cc:237] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:19.613804 4087 catalog_manager.cc:5484] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71). New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:19.625742 9953 raft_consensus.cc:482] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:19.626013 9953 raft_consensus.cc:504] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:19.627224 9953 leader_election.cc:288] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:19.628458 4363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9025370d3f44514a1430375e27c0239" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
W20240501 20:10:19.629590 4169 leader_election.cc:341] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:19.630010 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9025370d3f44514a1430375e27c0239" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:19.630333 7022 raft_consensus.cc:2454] T c9025370d3f44514a1430375e27c0239 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:19.631484 4169 leader_election.cc:302] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dca57823003e4729b769ca12367d7530, ef0a4ada548446b3a531cb39a3c4e38d; no voters: 10b47782558c41ee91f5e164a372049a
I20240501 20:10:19.632671 9953 raft_consensus.cc:2781] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:19.632843 9953 raft_consensus.cc:482] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:19.632949 9953 raft_consensus.cc:3037] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:19.634641 4087 catalog_manager.cc:5484] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:19.639482 9953 raft_consensus.cc:504] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:19.654330 4087 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:19.656491 4363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9025370d3f44514a1430375e27c0239" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
W20240501 20:10:19.657421 4169 leader_election.cc:341] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:19.659626 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9025370d3f44514a1430375e27c0239" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:19.659960 7022 raft_consensus.cc:3037] T c9025370d3f44514a1430375e27c0239 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:19.664695 7022 raft_consensus.cc:2454] T c9025370d3f44514a1430375e27c0239 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:19.665241 4169 leader_election.cc:302] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dca57823003e4729b769ca12367d7530, ef0a4ada548446b3a531cb39a3c4e38d; no voters: 10b47782558c41ee91f5e164a372049a
I20240501 20:10:19.665647 9802 raft_consensus.cc:2781] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:19.665846 9802 raft_consensus.cc:686] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:10:19.666182 9802 consensus_queue.cc:237] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:19.668653 9953 leader_election.cc:288] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:19.680409 10064 tablet.cc:1722] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:19.680928 10064 tablet.cc:1620] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:19.681538 10066 tablet.cc:1722] T 58afedbe61974f4c81e80b4648c18f5e P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:19.682224 10066 tablet.cc:1620] T 58afedbe61974f4c81e80b4648c18f5e P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:19.689492 10067 tablet.cc:1722] T 58afedbe61974f4c81e80b4648c18f5e P 3e974c03d3c546f9ab6c91e762a5426b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:19.690124 10067 tablet.cc:1620] T 58afedbe61974f4c81e80b4648c18f5e P 3e974c03d3c546f9ab6c91e762a5426b: MemRowSet was empty: no flush needed.
I20240501 20:10:19.693125 10056 tablet_bootstrap.cc:492] T c9025370d3f44514a1430375e27c0239 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:10:19.695880 10056 ts_tablet_manager.cc:1367] T c9025370d3f44514a1430375e27c0239 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.135s user 0.036s sys 0.053s
I20240501 20:10:19.698525 10056 raft_consensus.cc:348] T c9025370d3f44514a1430375e27c0239 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:19.701256 4089 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=9b6544bc81b445938dcf75f7ae09cd87] alter complete (version 2)
I20240501 20:10:19.702883 10056 raft_consensus.cc:374] T c9025370d3f44514a1430375e27c0239 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:19.702982 10056 raft_consensus.cc:729] T c9025370d3f44514a1430375e27c0239 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:10:19.703217 10056 consensus_queue.cc:260] T c9025370d3f44514a1430375e27c0239 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:19.705699 4087 catalog_manager.cc:5484] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:19.709848 10056 ts_tablet_manager.cc:1398] T c9025370d3f44514a1430375e27c0239 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240501 20:10:19.730883 9744 raft_consensus.cc:482] T c9025370d3f44514a1430375e27c0239 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:19.731139 9744 raft_consensus.cc:504] T c9025370d3f44514a1430375e27c0239 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:19.731928 9744 leader_election.cc:288] T c9025370d3f44514a1430375e27c0239 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:19.732795 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9025370d3f44514a1430375e27c0239" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:19.733646 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9025370d3f44514a1430375e27c0239" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:19.733927 7022 raft_consensus.cc:2379] T c9025370d3f44514a1430375e27c0239 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 10b47782558c41ee91f5e164a372049a in current term 1: Already voted for candidate dca57823003e4729b769ca12367d7530 in this term.
I20240501 20:10:19.734529 4301 leader_election.cc:302] T c9025370d3f44514a1430375e27c0239 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 10b47782558c41ee91f5e164a372049a; no voters: dca57823003e4729b769ca12367d7530, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:19.734994 9744 raft_consensus.cc:3037] T c9025370d3f44514a1430375e27c0239 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:19.737720 9744 raft_consensus.cc:2726] T c9025370d3f44514a1430375e27c0239 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:19.807670 4087 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:19.816700 10064 tablet.cc:1722] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:19.817546 10066 tablet.cc:1722] T 58afedbe61974f4c81e80b4648c18f5e P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:19.818022 10067 tablet.cc:1722] T 58afedbe61974f4c81e80b4648c18f5e P 3e974c03d3c546f9ab6c91e762a5426b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:19.828059 4092 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table_new_name [id=9b6544bc81b445938dcf75f7ae09cd87] alter complete (version 3)
I20240501 20:10:19.843302 4233 consensus_queue.cc:237] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:19.845535 7416 raft_consensus.cc:1261] T 264d3fcd3fae4acd8d76306c0f32ed72 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:19.845852 4169 consensus_peers.cc:485] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 264d3fcd3fae4acd8d76306c0f32ed72. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:19.846099 7155 raft_consensus.cc:1261] T 264d3fcd3fae4acd8d76306c0f32ed72 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:19.846076 10078 consensus_queue.cc:1035] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:19.846577 10078 consensus_queue.cc:1035] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:19.849233 8785 raft_consensus.cc:2932] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:19.849974 7416 raft_consensus.cc:2932] T 264d3fcd3fae4acd8d76306c0f32ed72 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:19.850476 7155 raft_consensus.cc:2932] T 264d3fcd3fae4acd8d76306c0f32ed72 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:19.856971 4087 catalog_manager.cc:5484] T 264d3fcd3fae4acd8d76306c0f32ed72 P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:19.878901 4233 consensus_queue.cc:237] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:19.881528 4496 raft_consensus.cc:1261] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:19.882071 7285 raft_consensus.cc:1261] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:19.882144 9802 consensus_queue.cc:1035] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:19.882905 9802 consensus_queue.cc:1035] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 20:10:19.883213 4169 consensus_peers.cc:485] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dfaa3794205d482c90aa4f860fa99cf3. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:19.886001 8785 raft_consensus.cc:2932] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:19.887032 7285 raft_consensus.cc:2932] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:19.893473 4089 catalog_manager.cc:5484] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 6 to 7, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:19.901536 4498 raft_consensus.cc:2932] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:19.927631 7416 consensus_queue.cc:237] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:19.930742 4089 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:19.930948 4089 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
W20240501 20:10:19.931015 7356 consensus_peers.cc:485] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 -> Peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Couldn't send request to peer 1d34e18a1d394d3f91c9f50d1c7bea4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 129322eab07149d2a6fb7eef96100e7a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:19.931257 7022 raft_consensus.cc:1261] T 129322eab07149d2a6fb7eef96100e7a P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:19.931685 7284 raft_consensus.cc:1261] T 129322eab07149d2a6fb7eef96100e7a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:19.931859 9660 consensus_queue.cc:1035] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:19.933782 9660 consensus_queue.cc:1035] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:19.934734 4089 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 58afedbe61974f4c81e80b4648c18f5e
I20240501 20:10:19.935870 7002 tablet_service.cc:1514] Processing DeleteTablet for tablet 58afedbe61974f4c81e80b4648c18f5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:19 UTC) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:19.936249 4477 tablet_service.cc:1514] Processing DeleteTablet for tablet 58afedbe61974f4c81e80b4648c18f5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:19 UTC) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:19.938271 9660 raft_consensus.cc:2932] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:19.939265 10105 tablet_replica.cc:330] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d: stopping tablet replica
I20240501 20:10:19.939502 10105 raft_consensus.cc:2229] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:19.939823 10105 raft_consensus.cc:2258] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:19.940429 7267 tablet_service.cc:1514] Processing DeleteTablet for tablet 58afedbe61974f4c81e80b4648c18f5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:19 UTC) from {username='slave'} at 127.0.0.1:54166
I20240501 20:10:19.940623 10105 ts_tablet_manager.cc:1875] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:19.941203 10107 tablet_replica.cc:330] T 58afedbe61974f4c81e80b4648c18f5e P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:19.941409 10107 raft_consensus.cc:2229] T 58afedbe61974f4c81e80b4648c18f5e P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:19.941695 10107 raft_consensus.cc:2258] T 58afedbe61974f4c81e80b4648c18f5e P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:19.941607 7022 raft_consensus.cc:2932] T 129322eab07149d2a6fb7eef96100e7a P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:19.954118 4089 catalog_manager.cc:5484] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:19.958786 10105 ts_tablet_manager.cc:1888] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:19.958961 10105 log.cc:1196] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/wals/58afedbe61974f4c81e80b4648c18f5e
I20240501 20:10:19.959439 10105 ts_tablet_manager.cc:1909] T 58afedbe61974f4c81e80b4648c18f5e P ef0a4ada548446b3a531cb39a3c4e38d: Deleting consensus metadata
I20240501 20:10:19.960477 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet 58afedbe61974f4c81e80b4648c18f5e (table rebalancer_dynamic_table_new_name [id=9b6544bc81b445938dcf75f7ae09cd87]) successfully deleted
I20240501 20:10:19.966943 10107 ts_tablet_manager.cc:1875] T 58afedbe61974f4c81e80b4648c18f5e P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:19.971266 10106 tablet_replica.cc:330] T 58afedbe61974f4c81e80b4648c18f5e P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:19.971716 10106 raft_consensus.cc:2229] T 58afedbe61974f4c81e80b4648c18f5e P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:19.972074 10106 raft_consensus.cc:2258] T 58afedbe61974f4c81e80b4648c18f5e P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:19.973431 10106 ts_tablet_manager.cc:1875] T 58afedbe61974f4c81e80b4648c18f5e P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:19.979015 10107 ts_tablet_manager.cc:1888] T 58afedbe61974f4c81e80b4648c18f5e P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:19.979300 10107 log.cc:1196] T 58afedbe61974f4c81e80b4648c18f5e P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/58afedbe61974f4c81e80b4648c18f5e
I20240501 20:10:19.979947 10107 ts_tablet_manager.cc:1909] T 58afedbe61974f4c81e80b4648c18f5e P bceeae1c79964acabdae8ff771bf0c97: Deleting consensus metadata
I20240501 20:10:19.988165 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet 58afedbe61974f4c81e80b4648c18f5e (table rebalancer_dynamic_table_new_name [id=9b6544bc81b445938dcf75f7ae09cd87]) successfully deleted
I20240501 20:10:19.996217 7285 raft_consensus.cc:2932] T 129322eab07149d2a6fb7eef96100e7a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:20.001598 10106 ts_tablet_manager.cc:1888] T 58afedbe61974f4c81e80b4648c18f5e P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:20.001897 10106 log.cc:1196] T 58afedbe61974f4c81e80b4648c18f5e P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/58afedbe61974f4c81e80b4648c18f5e
I20240501 20:10:20.004938 10106 ts_tablet_manager.cc:1909] T 58afedbe61974f4c81e80b4648c18f5e P 3e974c03d3c546f9ab6c91e762a5426b: Deleting consensus metadata
I20240501 20:10:20.006352 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet 58afedbe61974f4c81e80b4648c18f5e (table rebalancer_dynamic_table_new_name [id=9b6544bc81b445938dcf75f7ae09cd87]) successfully deleted
I20240501 20:10:20.023308 7285 consensus_queue.cc:237] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:20.025820 7419 raft_consensus.cc:1261] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:20.026084 7222 consensus_peers.cc:485] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c1e7ac38dc9c44d29e2d8127904407eb. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:20.026265 7155 raft_consensus.cc:1261] T c1e7ac38dc9c44d29e2d8127904407eb P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:20.026479 10116 consensus_queue.cc:1035] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:20.026830 9956 consensus_queue.cc:1035] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:20.039340 4087 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:20.040122 10075 raft_consensus.cc:2932] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:20.040733 7419 raft_consensus.cc:2932] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:20.048635 7153 raft_consensus.cc:2932] T c1e7ac38dc9c44d29e2d8127904407eb P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:20.050318 4088 catalog_manager.cc:5484] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:20.056345 7002 tablet_service.cc:1467] Processing CreateTablet for tablet 8a81cfcb50fb442a834d6f9de70bb0ca (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ba157bafcad84545ad579f1f1d68ba4a]), partition=
I20240501 20:10:20.056357 7267 tablet_service.cc:1467] Processing CreateTablet for tablet 8a81cfcb50fb442a834d6f9de70bb0ca (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ba157bafcad84545ad579f1f1d68ba4a]), partition=
I20240501 20:10:20.056712 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a81cfcb50fb442a834d6f9de70bb0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:20.056768 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a81cfcb50fb442a834d6f9de70bb0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:20.059983 7133 tablet_service.cc:1467] Processing CreateTablet for tablet 8a81cfcb50fb442a834d6f9de70bb0ca (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ba157bafcad84545ad579f1f1d68ba4a]), partition=
I20240501 20:10:20.060308 7133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a81cfcb50fb442a834d6f9de70bb0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:20.069000 9940 consensus_queue.cc:1035] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:20.078895 10124 tablet_bootstrap.cc:492] T 8a81cfcb50fb442a834d6f9de70bb0ca P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:20.080369 9319 consensus_queue.cc:1035] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:20.080718 10124 tablet_bootstrap.cc:654] T 8a81cfcb50fb442a834d6f9de70bb0ca P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:20.082319 10123 tablet_bootstrap.cc:492] T 8a81cfcb50fb442a834d6f9de70bb0ca P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:20.084022 10123 tablet_bootstrap.cc:654] T 8a81cfcb50fb442a834d6f9de70bb0ca P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:20.085426 10125 tablet_bootstrap.cc:492] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:20.087327 10123 tablet_bootstrap.cc:492] T 8a81cfcb50fb442a834d6f9de70bb0ca P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:20.087483 10123 ts_tablet_manager.cc:1367] T 8a81cfcb50fb442a834d6f9de70bb0ca P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 20:10:20.088585 10123 raft_consensus.cc:348] T 8a81cfcb50fb442a834d6f9de70bb0ca P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:20.088876 10123 raft_consensus.cc:374] T 8a81cfcb50fb442a834d6f9de70bb0ca P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:20.088954 10123 raft_consensus.cc:729] T 8a81cfcb50fb442a834d6f9de70bb0ca P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:20.089254 10123 consensus_queue.cc:260] T 8a81cfcb50fb442a834d6f9de70bb0ca P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:20.090097 10123 ts_tablet_manager.cc:1398] T 8a81cfcb50fb442a834d6f9de70bb0ca P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:10:20.093887 10125 tablet_bootstrap.cc:654] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:20.105981 10125 tablet_bootstrap.cc:492] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:20.106287 10125 ts_tablet_manager.cc:1367] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.000s
I20240501 20:10:20.107205 10125 raft_consensus.cc:348] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:20.107493 10125 raft_consensus.cc:374] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:20.107668 10125 raft_consensus.cc:729] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:20.108067 10125 consensus_queue.cc:260] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:20.108783 10125 ts_tablet_manager.cc:1398] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:20.112712 10075 raft_consensus.cc:482] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:20.112886 10075 raft_consensus.cc:504] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:20.113509 10075 leader_election.cc:288] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:20.114651 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a81cfcb50fb442a834d6f9de70bb0ca" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:20.114876 7022 raft_consensus.cc:2454] T 8a81cfcb50fb442a834d6f9de70bb0ca P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:20.115543 7222 leader_election.cc:302] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:20.116040 10116 raft_consensus.cc:2781] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:20.116171 10116 raft_consensus.cc:482] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:20.116248 10116 raft_consensus.cc:3037] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:20.116247 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a81cfcb50fb442a834d6f9de70bb0ca" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
W20240501 20:10:20.117019 7220 leader_election.cc:341] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:20.119200 10116 raft_consensus.cc:504] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:20.119498 10124 tablet_bootstrap.cc:492] T 8a81cfcb50fb442a834d6f9de70bb0ca P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:20.119673 10124 ts_tablet_manager.cc:1367] T 8a81cfcb50fb442a834d6f9de70bb0ca P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.000s
I20240501 20:10:20.121264 10124 raft_consensus.cc:348] T 8a81cfcb50fb442a834d6f9de70bb0ca P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:20.121516 10124 raft_consensus.cc:374] T 8a81cfcb50fb442a834d6f9de70bb0ca P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:20.121619 10124 raft_consensus.cc:729] T 8a81cfcb50fb442a834d6f9de70bb0ca P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:20.121940 10124 consensus_queue.cc:260] T 8a81cfcb50fb442a834d6f9de70bb0ca P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:20.122582 10075 consensus_queue.cc:1035] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:20.124120 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a81cfcb50fb442a834d6f9de70bb0ca" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:20.124323 7022 raft_consensus.cc:3037] T 8a81cfcb50fb442a834d6f9de70bb0ca P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:20.125209 10124 ts_tablet_manager.cc:1398] T 8a81cfcb50fb442a834d6f9de70bb0ca P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:10:20.127225 7022 raft_consensus.cc:2454] T 8a81cfcb50fb442a834d6f9de70bb0ca P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:20.127974 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a81cfcb50fb442a834d6f9de70bb0ca" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:20.128201 7155 raft_consensus.cc:3037] T 8a81cfcb50fb442a834d6f9de70bb0ca P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:20.129503 10116 leader_election.cc:288] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:20.130137 7222 leader_election.cc:302] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:20.131404 7155 raft_consensus.cc:2454] T 8a81cfcb50fb442a834d6f9de70bb0ca P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:20.132160 9953 consensus_queue.cc:1035] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:20.132730 10116 raft_consensus.cc:2781] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:20.132930 10116 raft_consensus.cc:686] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Becoming Leader. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Running, Role: LEADER
I20240501 20:10:20.133406 10116 consensus_queue.cc:237] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:20.140084 7022 consensus_queue.cc:237] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:20.145628 4089 catalog_manager.cc:5484] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71). New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:20.150774 7155 raft_consensus.cc:1261] T 79af181b5f014a40a4c123b47a720dec P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:20.151317 4367 raft_consensus.cc:1261] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:20.151651 9800 consensus_queue.cc:1035] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:20.155606 6957 consensus_peers.cc:485] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79af181b5f014a40a4c123b47a720dec. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:20.155965 9800 consensus_queue.cc:1035] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:20.158471 9956 consensus_queue.cc:1035] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:20.162423 10063 raft_consensus.cc:2932] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:20.165555 4367 raft_consensus.cc:2932] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:20.177141 9953 consensus_queue.cc:1035] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:20.190656 4087 catalog_manager.cc:5484] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:20.206318 7153 raft_consensus.cc:2932] T 79af181b5f014a40a4c123b47a720dec P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:20.258775 4092 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:20.259660 10151 ts_tablet_manager.cc:902] T 264d3fcd3fae4acd8d76306c0f32ed72 P 3e974c03d3c546f9ab6c91e762a5426b: Initiating tablet copy from peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:20.265247 7022 raft_consensus.cc:1261] T 8a81cfcb50fb442a834d6f9de70bb0ca P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:20.265233 7155 raft_consensus.cc:1261] T 8a81cfcb50fb442a834d6f9de70bb0ca P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:20.265941 10075 consensus_queue.cc:1035] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:20.266245 9956 consensus_queue.cc:1035] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:20.279062 4092 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:20.293673 4347 tablet_service.cc:1467] Processing CreateTablet for tablet bec3cae7dc9241fd8eae761967932f6e (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=c333ed244e1f4842956fe44221ea0c02]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:20.294184 4347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec3cae7dc9241fd8eae761967932f6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:20.294723 4477 tablet_service.cc:1467] Processing CreateTablet for tablet 4613582180754f729afc8d0b44356e97 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=c333ed244e1f4842956fe44221ea0c02]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:20.294839 10151 tablet_copy_client.cc:308] T 264d3fcd3fae4acd8d76306c0f32ed72 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.65:42941
I20240501 20:10:20.295143 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4613582180754f729afc8d0b44356e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:20.299101 7400 tablet_service.cc:1467] Processing CreateTablet for tablet bec3cae7dc9241fd8eae761967932f6e (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=c333ed244e1f4842956fe44221ea0c02]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:20.299211 4346 tablet_service.cc:1467] Processing CreateTablet for tablet 4613582180754f729afc8d0b44356e97 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=c333ed244e1f4842956fe44221ea0c02]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:20.299491 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec3cae7dc9241fd8eae761967932f6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:20.299692 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4613582180754f729afc8d0b44356e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:20.301666 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 82260307259e4a9fb12231ff4abd79c5 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=c333ed244e1f4842956fe44221ea0c02]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:20.301697 4609 tablet_service.cc:1467] Processing CreateTablet for tablet bec3cae7dc9241fd8eae761967932f6e (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=c333ed244e1f4842956fe44221ea0c02]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:20.302120 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec3cae7dc9241fd8eae761967932f6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:20.302156 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82260307259e4a9fb12231ff4abd79c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:20.303050 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 4613582180754f729afc8d0b44356e97 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=c333ed244e1f4842956fe44221ea0c02]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:20.303413 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4613582180754f729afc8d0b44356e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:20.305647 4254 tablet_copy_service.cc:140] P dca57823003e4729b769ca12367d7530: Received BeginTabletCopySession request for tablet 264d3fcd3fae4acd8d76306c0f32ed72 from peer 3e974c03d3c546f9ab6c91e762a5426b ({username='slave'} at 127.29.219.67:51285)
I20240501 20:10:20.305964 4254 tablet_copy_service.cc:161] P dca57823003e4729b769ca12367d7530: Beginning new tablet copy session on tablet 264d3fcd3fae4acd8d76306c0f32ed72 from peer 3e974c03d3c546f9ab6c91e762a5426b at {username='slave'} at 127.29.219.67:51285: session id = 3e974c03d3c546f9ab6c91e762a5426b-264d3fcd3fae4acd8d76306c0f32ed72
I20240501 20:10:20.307745 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 82260307259e4a9fb12231ff4abd79c5 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=c333ed244e1f4842956fe44221ea0c02]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:20.308496 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82260307259e4a9fb12231ff4abd79c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:20.311120 4348 tablet_service.cc:1467] Processing CreateTablet for tablet 82260307259e4a9fb12231ff4abd79c5 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=c333ed244e1f4842956fe44221ea0c02]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:20.311563 4348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82260307259e4a9fb12231ff4abd79c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:20.318459 10164 ts_tablet_manager.cc:902] T dfaa3794205d482c90aa4f860fa99cf3 P ef0a4ada548446b3a531cb39a3c4e38d: Initiating tablet copy from peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:20.322286 10163 tablet_bootstrap.cc:492] T bec3cae7dc9241fd8eae761967932f6e P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:20.324224 10163 tablet_bootstrap.cc:654] T bec3cae7dc9241fd8eae761967932f6e P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:20.324677 10161 tablet_bootstrap.cc:492] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:20.324990 4254 tablet_copy_source_session.cc:205] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:20.325670 10162 tablet_bootstrap.cc:492] T bec3cae7dc9241fd8eae761967932f6e P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:20.327920 10161 tablet_bootstrap.cc:654] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:20.329540 10162 tablet_bootstrap.cc:654] T bec3cae7dc9241fd8eae761967932f6e P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:20.330756 10151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 264d3fcd3fae4acd8d76306c0f32ed72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:20.341159 10151 tablet_copy_client.cc:783] T 264d3fcd3fae4acd8d76306c0f32ed72 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:20.341532 10151 tablet_copy_client.cc:647] T 264d3fcd3fae4acd8d76306c0f32ed72 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:20.343791 10161 tablet_bootstrap.cc:492] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:10:20.345544 10162 tablet_bootstrap.cc:492] T bec3cae7dc9241fd8eae761967932f6e P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:20.345794 10161 ts_tablet_manager.cc:1367] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.004s
I20240501 20:10:20.346535 10161 raft_consensus.cc:348] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:20.346921 10162 ts_tablet_manager.cc:1367] T bec3cae7dc9241fd8eae761967932f6e P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20240501 20:10:20.347982 10162 raft_consensus.cc:348] T bec3cae7dc9241fd8eae761967932f6e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:20.348182 10166 tablet_bootstrap.cc:492] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:20.348382 10161 raft_consensus.cc:374] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:20.348546 10161 raft_consensus.cc:729] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:10:20.348889 10161 consensus_queue.cc:260] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:20.349232 10162 raft_consensus.cc:374] T bec3cae7dc9241fd8eae761967932f6e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:20.349357 10162 raft_consensus.cc:729] T bec3cae7dc9241fd8eae761967932f6e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:20.350131 10166 tablet_bootstrap.cc:654] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:20.350603 10161 ts_tablet_manager.cc:1398] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:10:20.350013 10162 consensus_queue.cc:260] T bec3cae7dc9241fd8eae761967932f6e P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:20.351920 10161 tablet_bootstrap.cc:492] T 82260307259e4a9fb12231ff4abd79c5 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:20.354436 10151 tablet_copy_client.cc:523] T 264d3fcd3fae4acd8d76306c0f32ed72 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:20.354777 10162 ts_tablet_manager.cc:1398] T bec3cae7dc9241fd8eae761967932f6e P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 20:10:20.356148 10163 tablet_bootstrap.cc:492] T bec3cae7dc9241fd8eae761967932f6e P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:20.356439 10163 ts_tablet_manager.cc:1367] T bec3cae7dc9241fd8eae761967932f6e P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.002s
I20240501 20:10:20.357403 10163 raft_consensus.cc:348] T bec3cae7dc9241fd8eae761967932f6e P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:20.357677 10163 raft_consensus.cc:374] T bec3cae7dc9241fd8eae761967932f6e P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:20.357816 10163 raft_consensus.cc:729] T bec3cae7dc9241fd8eae761967932f6e P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:20.360869 10163 consensus_queue.cc:260] T bec3cae7dc9241fd8eae761967932f6e P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:20.364367 10151 tablet_bootstrap.cc:492] T 264d3fcd3fae4acd8d76306c0f32ed72 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:20.364414 10163 ts_tablet_manager.cc:1398] T bec3cae7dc9241fd8eae761967932f6e P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240501 20:10:20.364806 10163 tablet_bootstrap.cc:492] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:20.365131 10162 tablet_bootstrap.cc:492] T 4613582180754f729afc8d0b44356e97 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:20.366307 10161 tablet_bootstrap.cc:654] T 82260307259e4a9fb12231ff4abd79c5 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:20.369809 4087 catalog_manager.cc:5576] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237) does not have the latest schema for tablet 8a81cfcb50fb442a834d6f9de70bb0ca (table rebalancer_dynamic_table [id=ba157bafcad84545ad579f1f1d68ba4a]). Expected version 1 got 0
I20240501 20:10:20.375140 10163 tablet_bootstrap.cc:654] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:20.376086 10162 tablet_bootstrap.cc:654] T 4613582180754f729afc8d0b44356e97 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:20.379421 4089 catalog_manager.cc:5576] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237) does not have the latest schema for tablet 8a81cfcb50fb442a834d6f9de70bb0ca (table rebalancer_dynamic_table [id=ba157bafcad84545ad579f1f1d68ba4a]). Expected version 1 got 0
I20240501 20:10:20.376789 10172 tablet.cc:1722] T 8a81cfcb50fb442a834d6f9de70bb0ca P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:20.375025 4365 consensus_queue.cc:237] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:20.385079 10171 tablet.cc:1722] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:20.385648 10171 tablet.cc:1620] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:20.386417 10172 tablet.cc:1620] T 8a81cfcb50fb442a834d6f9de70bb0ca P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:20.396492 4092 catalog_manager.cc:5576] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447) does not have the latest schema for tablet 8a81cfcb50fb442a834d6f9de70bb0ca (table rebalancer_dynamic_table [id=ba157bafcad84545ad579f1f1d68ba4a]). Expected version 1 got 0
I20240501 20:10:20.400262 10170 tablet_bootstrap.cc:492] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:20.407195 7019 raft_consensus.cc:1261] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:20.407749 7153 raft_consensus.cc:1261] T 5a7b2f01b5684df5800987591a390a17 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:20.407855 4301 consensus_peers.cc:485] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a7b2f01b5684df5800987591a390a17. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:20.408026 9744 consensus_queue.cc:1035] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:20.409155 9744 consensus_queue.cc:1035] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:20.411304 10163 tablet_bootstrap.cc:492] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:20.411584 10163 ts_tablet_manager.cc:1367] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.003s
I20240501 20:10:20.412081 4089 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=ba157bafcad84545ad579f1f1d68ba4a] alter complete (version 1)
I20240501 20:10:20.412536 10163 raft_consensus.cc:348] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:20.412882 10163 raft_consensus.cc:374] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:20.414731 10163 raft_consensus.cc:729] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:20.415207 10163 consensus_queue.cc:260] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:20.418956 10163 ts_tablet_manager.cc:1398] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240501 20:10:20.420091 10162 tablet_bootstrap.cc:492] T 4613582180754f729afc8d0b44356e97 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:20.420289 10162 ts_tablet_manager.cc:1367] T 4613582180754f729afc8d0b44356e97 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.003s
I20240501 20:10:20.421192 10162 raft_consensus.cc:348] T 4613582180754f729afc8d0b44356e97 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:20.421412 4087 catalog_manager.cc:5576] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321) does not have the latest schema for tablet 8a81cfcb50fb442a834d6f9de70bb0ca (table rebalancer_dynamic_table [id=ba157bafcad84545ad579f1f1d68ba4a]). Expected version 1 got 0
I20240501 20:10:20.422016 10162 raft_consensus.cc:374] T 4613582180754f729afc8d0b44356e97 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:20.422163 10162 raft_consensus.cc:729] T 4613582180754f729afc8d0b44356e97 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:20.422504 10162 consensus_queue.cc:260] T 4613582180754f729afc8d0b44356e97 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:20.423302 10162 ts_tablet_manager.cc:1398] T 4613582180754f729afc8d0b44356e97 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:10:20.425802 10161 tablet_bootstrap.cc:492] T 82260307259e4a9fb12231ff4abd79c5 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:10:20.426044 10161 ts_tablet_manager.cc:1367] T 82260307259e4a9fb12231ff4abd79c5 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.002s
I20240501 20:10:20.427176 10161 raft_consensus.cc:348] T 82260307259e4a9fb12231ff4abd79c5 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:20.427580 10161 raft_consensus.cc:374] T 82260307259e4a9fb12231ff4abd79c5 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:20.427726 10161 raft_consensus.cc:729] T 82260307259e4a9fb12231ff4abd79c5 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:10:20.428083 10161 consensus_queue.cc:260] T 82260307259e4a9fb12231ff4abd79c5 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:20.431260 10161 ts_tablet_manager.cc:1398] T 82260307259e4a9fb12231ff4abd79c5 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:10:20.438482 10183 tablet.cc:1722] T 8a81cfcb50fb442a834d6f9de70bb0ca P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:20.441570 10161 tablet_bootstrap.cc:492] T 4613582180754f729afc8d0b44356e97 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:20.442992 4089 catalog_manager.cc:5576] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321) does not have the latest schema for tablet 8a81cfcb50fb442a834d6f9de70bb0ca (table rebalancer_dynamic_table [id=ba157bafcad84545ad579f1f1d68ba4a]). Expected version 1 got 0
I20240501 20:10:20.444859 10161 tablet_bootstrap.cc:654] T 4613582180754f729afc8d0b44356e97 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:20.448388 10166 tablet_bootstrap.cc:492] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:20.448665 10166 ts_tablet_manager.cc:1367] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.101s user 0.005s sys 0.008s
I20240501 20:10:20.449740 10166 raft_consensus.cc:348] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:20.450053 10166 raft_consensus.cc:374] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:20.450253 10166 raft_consensus.cc:729] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:20.446261 10183 tablet.cc:1620] T 8a81cfcb50fb442a834d6f9de70bb0ca P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:10:20.450554 10166 consensus_queue.cc:260] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:20.451406 10166 ts_tablet_manager.cc:1398] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:20.451216 9744 raft_consensus.cc:2932] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:20.452363 10151 tablet_bootstrap.cc:492] T 264d3fcd3fae4acd8d76306c0f32ed72 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:20.452756 10151 tablet_bootstrap.cc:492] T 264d3fcd3fae4acd8d76306c0f32ed72 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap complete.
I20240501 20:10:20.453023 10151 ts_tablet_manager.cc:1367] T 264d3fcd3fae4acd8d76306c0f32ed72 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.089s user 0.007s sys 0.017s
I20240501 20:10:20.454985 7154 raft_consensus.cc:2932] T 5a7b2f01b5684df5800987591a390a17 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:20.453732 10151 raft_consensus.cc:348] T 264d3fcd3fae4acd8d76306c0f32ed72 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:20.457307 10151 raft_consensus.cc:729] T 264d3fcd3fae4acd8d76306c0f32ed72 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: LEARNER
I20240501 20:10:20.464607 4088 catalog_manager.cc:5484] T 5a7b2f01b5684df5800987591a390a17 P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:20.467785 10170 tablet_bootstrap.cc:654] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:20.476101 10170 tablet_bootstrap.cc:492] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:10:20.476363 10170 ts_tablet_manager.cc:1367] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.076s user 0.005s sys 0.002s
I20240501 20:10:20.493250 7022 raft_consensus.cc:2932] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:20.477298 10170 raft_consensus.cc:348] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:20.497236 10170 raft_consensus.cc:374] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:20.497555 10170 raft_consensus.cc:729] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:10:20.498147 10170 consensus_queue.cc:260] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:20.499469 10170 ts_tablet_manager.cc:1398] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20240501 20:10:20.515731 10164 tablet_copy_client.cc:308] T dfaa3794205d482c90aa4f860fa99cf3 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.65:42941
I20240501 20:10:20.517453 4254 tablet_copy_service.cc:140] P dca57823003e4729b769ca12367d7530: Received BeginTabletCopySession request for tablet dfaa3794205d482c90aa4f860fa99cf3 from peer ef0a4ada548446b3a531cb39a3c4e38d ({username='slave'} at 127.29.219.69:51289)
I20240501 20:10:20.517666 4254 tablet_copy_service.cc:161] P dca57823003e4729b769ca12367d7530: Beginning new tablet copy session on tablet dfaa3794205d482c90aa4f860fa99cf3 from peer ef0a4ada548446b3a531cb39a3c4e38d at {username='slave'} at 127.29.219.69:51289: session id = ef0a4ada548446b3a531cb39a3c4e38d-dfaa3794205d482c90aa4f860fa99cf3
I20240501 20:10:20.520486 4254 tablet_copy_source_session.cc:205] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:20.522082 10164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfaa3794205d482c90aa4f860fa99cf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:20.457624 10151 consensus_queue.cc:260] T 264d3fcd3fae4acd8d76306c0f32ed72 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:20.524734 4088 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:20.529548 10151 ts_tablet_manager.cc:1398] T 264d3fcd3fae4acd8d76306c0f32ed72 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.076s user 0.003s sys 0.001s
I20240501 20:10:20.532866 10190 ts_tablet_manager.cc:902] T 129322eab07149d2a6fb7eef96100e7a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Initiating tablet copy from peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:20.534525 10190 tablet_copy_client.cc:308] T 129322eab07149d2a6fb7eef96100e7a P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.72:39817
I20240501 20:10:20.544772 4254 tablet_copy_service.cc:342] P dca57823003e4729b769ca12367d7530: Request end of tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-264d3fcd3fae4acd8d76306c0f32ed72 received from {username='slave'} at 127.29.219.67:51285
I20240501 20:10:20.544999 4254 tablet_copy_service.cc:433] P dca57823003e4729b769ca12367d7530: ending tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-264d3fcd3fae4acd8d76306c0f32ed72 on tablet 264d3fcd3fae4acd8d76306c0f32ed72 with peer 3e974c03d3c546f9ab6c91e762a5426b
I20240501 20:10:20.552160 10161 tablet_bootstrap.cc:492] T 4613582180754f729afc8d0b44356e97 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:10:20.552342 10161 ts_tablet_manager.cc:1367] T 4613582180754f729afc8d0b44356e97 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.111s user 0.046s sys 0.034s
I20240501 20:10:20.553081 10161 raft_consensus.cc:348] T 4613582180754f729afc8d0b44356e97 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:20.553289 10161 raft_consensus.cc:374] T 4613582180754f729afc8d0b44356e97 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:20.553366 10161 raft_consensus.cc:729] T 4613582180754f729afc8d0b44356e97 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:10:20.553570 10161 consensus_queue.cc:260] T 4613582180754f729afc8d0b44356e97 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:20.554129 10161 ts_tablet_manager.cc:1398] T 4613582180754f729afc8d0b44356e97 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:20.554296 10171 tablet.cc:1722] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:20.554888 10171 tablet.cc:1620] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:20.556399 10183 tablet.cc:1722] T 8a81cfcb50fb442a834d6f9de70bb0ca P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:20.556479 10172 tablet.cc:1722] T 8a81cfcb50fb442a834d6f9de70bb0ca P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:20.556885 10183 tablet.cc:1620] T 8a81cfcb50fb442a834d6f9de70bb0ca P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:10:20.556950 10172 tablet.cc:1620] T 8a81cfcb50fb442a834d6f9de70bb0ca P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:20.557479 9953 raft_consensus.cc:482] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:20.557685 9953 raft_consensus.cc:504] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:20.558254 10164 tablet_copy_client.cc:783] T dfaa3794205d482c90aa4f860fa99cf3 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:20.558513 10164 tablet_copy_client.cc:647] T dfaa3794205d482c90aa4f860fa99cf3 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:20.558501 9953 leader_election.cc:288] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:20.560145 7416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82260307259e4a9fb12231ff4abd79c5" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:20.560397 7416 raft_consensus.cc:2454] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:20.560947 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82260307259e4a9fb12231ff4abd79c5" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:10:20.561244 4365 raft_consensus.cc:2454] T 82260307259e4a9fb12231ff4abd79c5 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:20.561968 4170 leader_election.cc:302] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:20.572655 4088 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=ba157bafcad84545ad579f1f1d68ba4a] alter complete (version 2)
I20240501 20:10:20.574560 9953 raft_consensus.cc:2781] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:20.574738 9953 raft_consensus.cc:482] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:20.574875 9953 raft_consensus.cc:3037] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:20.583204 9744 raft_consensus.cc:482] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:20.583454 9744 raft_consensus.cc:504] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:20.584404 9744 leader_election.cc:288] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:20.584553 10164 tablet_copy_client.cc:523] T dfaa3794205d482c90aa4f860fa99cf3 P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:20.584890 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec3cae7dc9241fd8eae761967932f6e" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:20.585095 4627 raft_consensus.cc:2454] T bec3cae7dc9241fd8eae761967932f6e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:10:20.585227 7441 tablet_copy_service.cc:140] P b9957d4086bc4a76b18ed9265a5739d5: Received BeginTabletCopySession request for tablet 129322eab07149d2a6fb7eef96100e7a from peer 1d34e18a1d394d3f91c9f50d1c7bea4e ({username='slave'} at 127.29.219.68:34551)
I20240501 20:10:20.585455 7441 tablet_copy_service.cc:161] P b9957d4086bc4a76b18ed9265a5739d5: Beginning new tablet copy session on tablet 129322eab07149d2a6fb7eef96100e7a from peer 1d34e18a1d394d3f91c9f50d1c7bea4e at {username='slave'} at 127.29.219.68:34551: session id = 1d34e18a1d394d3f91c9f50d1c7bea4e-129322eab07149d2a6fb7eef96100e7a
I20240501 20:10:20.585548 4302 leader_election.cc:302] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 1d34e18a1d394d3f91c9f50d1c7bea4e; no voters:
I20240501 20:10:20.586174 7416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec3cae7dc9241fd8eae761967932f6e" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:20.586403 7416 raft_consensus.cc:2454] T bec3cae7dc9241fd8eae761967932f6e P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:10:20.588716 7441 tablet_copy_source_session.cc:205] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:20.589644 9744 raft_consensus.cc:2781] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:20.589769 9744 raft_consensus.cc:482] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:20.589869 9744 raft_consensus.cc:3037] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:20.592234 9744 raft_consensus.cc:504] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:20.593006 9744 leader_election.cc:288] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Requested vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:20.593489 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec3cae7dc9241fd8eae761967932f6e" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:10:20.593677 4627 raft_consensus.cc:3037] T bec3cae7dc9241fd8eae761967932f6e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:20.592716 9953 raft_consensus.cc:504] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:20.597853 4363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82260307259e4a9fb12231ff4abd79c5" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:10:20.598060 4363 raft_consensus.cc:3037] T 82260307259e4a9fb12231ff4abd79c5 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:20.598799 7416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec3cae7dc9241fd8eae761967932f6e" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:20.599006 7416 raft_consensus.cc:3037] T bec3cae7dc9241fd8eae761967932f6e P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:20.600826 4363 raft_consensus.cc:2454] T 82260307259e4a9fb12231ff4abd79c5 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:20.601691 7416 raft_consensus.cc:2454] T bec3cae7dc9241fd8eae761967932f6e P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:10:20.602954 4169 leader_election.cc:302] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:20.603276 8785 raft_consensus.cc:2781] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:20.603410 8785 raft_consensus.cc:686] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:10:20.603397 10190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 129322eab07149d2a6fb7eef96100e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:20.603641 8785 consensus_queue.cc:237] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:20.605687 10164 tablet_bootstrap.cc:492] T dfaa3794205d482c90aa4f860fa99cf3 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:20.598558 4627 raft_consensus.cc:2454] T bec3cae7dc9241fd8eae761967932f6e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:10:20.607425 4302 leader_election.cc:302] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 1d34e18a1d394d3f91c9f50d1c7bea4e; no voters:
I20240501 20:10:20.607909 10144 raft_consensus.cc:2781] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:20.608055 10144 raft_consensus.cc:686] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Becoming Leader. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Running, Role: LEADER
I20240501 20:10:20.608326 10144 consensus_queue.cc:237] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:20.610895 10190 tablet_copy_client.cc:783] T 129322eab07149d2a6fb7eef96100e7a P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:20.611141 10190 tablet_copy_client.cc:647] T 129322eab07149d2a6fb7eef96100e7a P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:20.613870 10190 tablet_copy_client.cc:523] T 129322eab07149d2a6fb7eef96100e7a P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:20.617439 4087 catalog_manager.cc:5484] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a reported cstate change: term changed from 0 to 1, leader changed from <none> to 10b47782558c41ee91f5e164a372049a (127.29.219.66). New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:20.617981 10190 tablet_bootstrap.cc:492] T 129322eab07149d2a6fb7eef96100e7a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:20.621665 7416 consensus_queue.cc:237] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:20.629695 10151 ts_tablet_manager.cc:902] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b: Initiating tablet copy from peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:20.631578 9953 leader_election.cc:288] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:20.632143 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82260307259e4a9fb12231ff4abd79c5" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:20.632341 7419 raft_consensus.cc:3037] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
W20240501 20:10:20.636914 7356 consensus_peers.cc:485] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 -> Peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Couldn't send request to peer 1d34e18a1d394d3f91c9f50d1c7bea4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e31f8d1809244efae05ab9254c163ce. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:20.637030 7419 raft_consensus.cc:2454] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:20.638099 4234 raft_consensus.cc:1261] T 9e31f8d1809244efae05ab9254c163ce P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:20.638566 7285 raft_consensus.cc:1261] T 9e31f8d1809244efae05ab9254c163ce P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:20.638778 10046 consensus_queue.cc:1035] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:20.640064 10046 consensus_queue.cc:1035] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:20.640419 4088 catalog_manager.cc:5484] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:20.650156 10046 raft_consensus.cc:2932] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:20.654831 4234 raft_consensus.cc:2932] T 9e31f8d1809244efae05ab9254c163ce P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:20.656535 10202 ts_tablet_manager.cc:902] T c1e7ac38dc9c44d29e2d8127904407eb P 10b47782558c41ee91f5e164a372049a: Initiating tablet copy from peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:20.661815 4088 catalog_manager.cc:5484] T 9e31f8d1809244efae05ab9254c163ce P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:20.677279 10190 tablet_bootstrap.cc:492] T 129322eab07149d2a6fb7eef96100e7a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:20.677706 10190 tablet_bootstrap.cc:492] T 129322eab07149d2a6fb7eef96100e7a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap complete.
I20240501 20:10:20.677989 10190 ts_tablet_manager.cc:1367] T 129322eab07149d2a6fb7eef96100e7a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.060s user 0.016s sys 0.016s
I20240501 20:10:20.680603 10190 raft_consensus.cc:348] T 129322eab07149d2a6fb7eef96100e7a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:20.680850 10190 raft_consensus.cc:729] T 129322eab07149d2a6fb7eef96100e7a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: LEARNER
I20240501 20:10:20.681033 10190 consensus_queue.cc:260] T 129322eab07149d2a6fb7eef96100e7a P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:20.686491 10190 ts_tablet_manager.cc:1398] T 129322eab07149d2a6fb7eef96100e7a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20240501 20:10:20.693348 7441 tablet_copy_service.cc:342] P b9957d4086bc4a76b18ed9265a5739d5: Request end of tablet copy session 1d34e18a1d394d3f91c9f50d1c7bea4e-129322eab07149d2a6fb7eef96100e7a received from {username='slave'} at 127.29.219.68:34551
I20240501 20:10:20.693576 7441 tablet_copy_service.cc:433] P b9957d4086bc4a76b18ed9265a5739d5: ending tablet copy session 1d34e18a1d394d3f91c9f50d1c7bea4e-129322eab07149d2a6fb7eef96100e7a on tablet 129322eab07149d2a6fb7eef96100e7a with peer 1d34e18a1d394d3f91c9f50d1c7bea4e
I20240501 20:10:20.698050 4087 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:20.714828 10202 tablet_copy_client.cc:308] T c1e7ac38dc9c44d29e2d8127904407eb P 10b47782558c41ee91f5e164a372049a: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.71:39447
I20240501 20:10:20.716329 7308 tablet_copy_service.cc:140] P bceeae1c79964acabdae8ff771bf0c97: Received BeginTabletCopySession request for tablet c1e7ac38dc9c44d29e2d8127904407eb from peer 10b47782558c41ee91f5e164a372049a ({username='slave'} at 127.29.219.66:32801)
I20240501 20:10:20.716506 7308 tablet_copy_service.cc:161] P bceeae1c79964acabdae8ff771bf0c97: Beginning new tablet copy session on tablet c1e7ac38dc9c44d29e2d8127904407eb from peer 10b47782558c41ee91f5e164a372049a at {username='slave'} at 127.29.219.66:32801: session id = 10b47782558c41ee91f5e164a372049a-c1e7ac38dc9c44d29e2d8127904407eb
I20240501 20:10:20.718832 10171 tablet.cc:1722] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:20.719434 7308 tablet_copy_source_session.cc:205] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:20.721272 7287 raft_consensus.cc:2932] T 9e31f8d1809244efae05ab9254c163ce P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:20.728785 10172 tablet.cc:1722] T 8a81cfcb50fb442a834d6f9de70bb0ca P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:20.738229 10183 tablet.cc:1722] T 8a81cfcb50fb442a834d6f9de70bb0ca P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:20.743439 10202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1e7ac38dc9c44d29e2d8127904407eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:20.748531 4089 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table_new_name [id=ba157bafcad84545ad579f1f1d68ba4a] alter complete (version 3)
I20240501 20:10:20.767135 10151 tablet_copy_client.cc:308] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.69:46237
I20240501 20:10:20.768100 7042 tablet_copy_service.cc:140] P ef0a4ada548446b3a531cb39a3c4e38d: Received BeginTabletCopySession request for tablet 79af181b5f014a40a4c123b47a720dec from peer 3e974c03d3c546f9ab6c91e762a5426b ({username='slave'} at 127.29.219.67:45609)
I20240501 20:10:20.768301 7042 tablet_copy_service.cc:161] P ef0a4ada548446b3a531cb39a3c4e38d: Beginning new tablet copy session on tablet 79af181b5f014a40a4c123b47a720dec from peer 3e974c03d3c546f9ab6c91e762a5426b at {username='slave'} at 127.29.219.67:45609: session id = 3e974c03d3c546f9ab6c91e762a5426b-79af181b5f014a40a4c123b47a720dec
I20240501 20:10:20.771629 7042 tablet_copy_source_session.cc:205] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:20.773407 10202 tablet_copy_client.cc:783] T c1e7ac38dc9c44d29e2d8127904407eb P 10b47782558c41ee91f5e164a372049a: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:20.773487 10151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79af181b5f014a40a4c123b47a720dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:20.773792 10202 tablet_copy_client.cc:647] T c1e7ac38dc9c44d29e2d8127904407eb P 10b47782558c41ee91f5e164a372049a: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:20.774719 4496 consensus_queue.cc:237] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:20.775612 10202 tablet_copy_client.cc:523] T c1e7ac38dc9c44d29e2d8127904407eb P 10b47782558c41ee91f5e164a372049a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:20.785204 4498 raft_consensus.cc:1203] T 264d3fcd3fae4acd8d76306c0f32ed72 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 20:10:20.788628 4434 consensus_peers.cc:485] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61af956bc72b4e3e994a7daf58d1a699. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:20.789706 7416 raft_consensus.cc:1261] T 61af956bc72b4e3e994a7daf58d1a699 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:20.790339 9940 consensus_queue.cc:1035] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:20.794979 9319 raft_consensus.cc:482] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:20.795161 9319 raft_consensus.cc:504] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:20.795925 9319 leader_election.cc:288] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:20.796312 9940 raft_consensus.cc:2932] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:20.796911 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4613582180754f729afc8d0b44356e97" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:20.797113 4627 raft_consensus.cc:2454] T 4613582180754f729afc8d0b44356e97 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:10:20.797204 7416 raft_consensus.cc:2932] T 61af956bc72b4e3e994a7daf58d1a699 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:20.797629 4435 leader_election.cc:302] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:10:20.797998 9319 raft_consensus.cc:2781] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:20.798122 9319 raft_consensus.cc:482] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:20.798203 9319 raft_consensus.cc:3037] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:20.798566 10202 tablet_bootstrap.cc:492] T c1e7ac38dc9c44d29e2d8127904407eb P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:20.800693 10151 tablet_copy_client.cc:783] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:20.800943 10151 tablet_copy_client.cc:647] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:20.800892 9319 raft_consensus.cc:504] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
W20240501 20:10:20.801429 7353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.70:36321, user_credentials={real_user=slave}} blocked reactor thread for 51702.6us
I20240501 20:10:20.801474 9319 leader_election.cc:288] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:20.802501 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4613582180754f729afc8d0b44356e97" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:10:20.802719 4627 raft_consensus.cc:3037] T 4613582180754f729afc8d0b44356e97 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:20.805344 4627 raft_consensus.cc:2454] T 4613582180754f729afc8d0b44356e97 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:10:20.809001 4088 catalog_manager.cc:5484] T 61af956bc72b4e3e994a7daf58d1a699 P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:20.819348 4363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4613582180754f729afc8d0b44356e97" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:10:20.819617 4363 raft_consensus.cc:2454] T 4613582180754f729afc8d0b44356e97 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:10:20.820320 4363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4613582180754f729afc8d0b44356e97" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:10:20.820519 4363 raft_consensus.cc:3037] T 4613582180754f729afc8d0b44356e97 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:20.820680 4435 leader_election.cc:302] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:10:20.821110 10228 raft_consensus.cc:2781] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:20.821797 10228 raft_consensus.cc:686] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:10:20.822283 10228 consensus_queue.cc:237] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:20.822788 4363 raft_consensus.cc:2454] T 4613582180754f729afc8d0b44356e97 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:10:20.823896 4234 raft_consensus.cc:1261] T 61af956bc72b4e3e994a7daf58d1a699 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:20.833791 10151 tablet_copy_client.cc:523] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:20.834167 10231 consensus_queue.cc:1035] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:20.839658 10151 tablet_bootstrap.cc:492] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:20.840021 4087 catalog_manager.cc:5484] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:20.854101 4234 raft_consensus.cc:2932] T 61af956bc72b4e3e994a7daf58d1a699 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:20.857266 4087 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:20.857524 4087 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:20.861204 4087 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 8a81cfcb50fb442a834d6f9de70bb0ca
I20240501 20:10:20.862633 7002 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a81cfcb50fb442a834d6f9de70bb0ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:20 UTC) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:20.862962 10202 tablet_bootstrap.cc:492] T c1e7ac38dc9c44d29e2d8127904407eb P 10b47782558c41ee91f5e164a372049a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:20.863031 7133 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a81cfcb50fb442a834d6f9de70bb0ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:20 UTC) from {username='slave'} at 127.0.0.1:52036
I20240501 20:10:20.863358 10202 tablet_bootstrap.cc:492] T c1e7ac38dc9c44d29e2d8127904407eb P 10b47782558c41ee91f5e164a372049a: Bootstrap complete.
I20240501 20:10:20.864180 10202 ts_tablet_manager.cc:1367] T c1e7ac38dc9c44d29e2d8127904407eb P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.066s user 0.008s sys 0.003s
I20240501 20:10:20.864306 7267 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a81cfcb50fb442a834d6f9de70bb0ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:20 UTC) from {username='slave'} at 127.0.0.1:54166
I20240501 20:10:20.865337 10202 raft_consensus.cc:348] T c1e7ac38dc9c44d29e2d8127904407eb P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:20.865641 10202 raft_consensus.cc:729] T c1e7ac38dc9c44d29e2d8127904407eb P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: LEARNER
I20240501 20:10:20.865840 10202 consensus_queue.cc:260] T c1e7ac38dc9c44d29e2d8127904407eb P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:20.866344 10202 ts_tablet_manager.cc:1398] T c1e7ac38dc9c44d29e2d8127904407eb P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:20.869135 7308 tablet_copy_service.cc:342] P bceeae1c79964acabdae8ff771bf0c97: Request end of tablet copy session 10b47782558c41ee91f5e164a372049a-c1e7ac38dc9c44d29e2d8127904407eb received from {username='slave'} at 127.29.219.66:32801
I20240501 20:10:20.869277 7308 tablet_copy_service.cc:433] P bceeae1c79964acabdae8ff771bf0c97: ending tablet copy session 10b47782558c41ee91f5e164a372049a-c1e7ac38dc9c44d29e2d8127904407eb on tablet c1e7ac38dc9c44d29e2d8127904407eb with peer 10b47782558c41ee91f5e164a372049a
I20240501 20:10:20.893138 10151 tablet_bootstrap.cc:492] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:20.893628 10151 tablet_bootstrap.cc:492] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap complete.
I20240501 20:10:20.895299 10151 ts_tablet_manager.cc:1367] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.011s
I20240501 20:10:20.896266 10151 raft_consensus.cc:348] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:20.896589 10151 raft_consensus.cc:729] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: LEARNER
I20240501 20:10:20.896860 10151 consensus_queue.cc:260] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:20.899339 10151 ts_tablet_manager.cc:1398] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:10:20.900135 7042 tablet_copy_service.cc:342] P ef0a4ada548446b3a531cb39a3c4e38d: Request end of tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-79af181b5f014a40a4c123b47a720dec received from {username='slave'} at 127.29.219.67:45609
I20240501 20:10:20.900285 7042 tablet_copy_service.cc:433] P ef0a4ada548446b3a531cb39a3c4e38d: ending tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-79af181b5f014a40a4c123b47a720dec on tablet 79af181b5f014a40a4c123b47a720dec with peer 3e974c03d3c546f9ab6c91e762a5426b
I20240501 20:10:20.904856 10241 tablet_replica.cc:330] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:20.905390 10241 raft_consensus.cc:2229] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:20.907533 10245 ts_tablet_manager.cc:902] T 5a7b2f01b5684df5800987591a390a17 P dca57823003e4729b769ca12367d7530: Initiating tablet copy from peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:20.909526 10241 raft_consensus.cc:2258] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:20.910542 10241 ts_tablet_manager.cc:1875] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:20.913008 10239 tablet_replica.cc:330] T 8a81cfcb50fb442a834d6f9de70bb0ca P ef0a4ada548446b3a531cb39a3c4e38d: stopping tablet replica
I20240501 20:10:20.913305 10239 raft_consensus.cc:2229] T 8a81cfcb50fb442a834d6f9de70bb0ca P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:20.913575 10239 raft_consensus.cc:2258] T 8a81cfcb50fb442a834d6f9de70bb0ca P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:20.914498 10239 ts_tablet_manager.cc:1875] T 8a81cfcb50fb442a834d6f9de70bb0ca P ef0a4ada548446b3a531cb39a3c4e38d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:20.915036 10240 tablet_replica.cc:330] T 8a81cfcb50fb442a834d6f9de70bb0ca P db0742ee3860468590eefef6ae6074fd: stopping tablet replica
I20240501 20:10:20.915513 10240 raft_consensus.cc:2229] T 8a81cfcb50fb442a834d6f9de70bb0ca P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:20.915771 10240 raft_consensus.cc:2258] T 8a81cfcb50fb442a834d6f9de70bb0ca P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:20.917498 10241 ts_tablet_manager.cc:1888] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:20.917714 10241 log.cc:1196] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/8a81cfcb50fb442a834d6f9de70bb0ca
I20240501 20:10:20.918355 10241 ts_tablet_manager.cc:1909] T 8a81cfcb50fb442a834d6f9de70bb0ca P bceeae1c79964acabdae8ff771bf0c97: Deleting consensus metadata
I20240501 20:10:20.919478 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet 8a81cfcb50fb442a834d6f9de70bb0ca (table rebalancer_dynamic_table_new_name [id=ba157bafcad84545ad579f1f1d68ba4a]) successfully deleted
I20240501 20:10:20.923264 10240 ts_tablet_manager.cc:1875] T 8a81cfcb50fb442a834d6f9de70bb0ca P db0742ee3860468590eefef6ae6074fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:20.924573 4363 consensus_queue.cc:237] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:20.934062 4496 raft_consensus.cc:1261] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:20.934717 9744 consensus_queue.cc:1035] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:20.934952 7287 raft_consensus.cc:1261] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:20.935560 9744 consensus_queue.cc:1035] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 20:10:20.937506 4299 consensus_peers.cc:485] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a -> Peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Couldn't send request to peer db0742ee3860468590eefef6ae6074fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c66514deec5744da832bc030a582c46a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:20.953140 10164 tablet_bootstrap.cc:492] T dfaa3794205d482c90aa4f860fa99cf3 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:20.953276 9744 raft_consensus.cc:2932] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:20.955395 10164 tablet_bootstrap.cc:492] T dfaa3794205d482c90aa4f860fa99cf3 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap complete.
I20240501 20:10:20.956481 4496 raft_consensus.cc:2932] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:20.957939 10164 ts_tablet_manager.cc:1367] T dfaa3794205d482c90aa4f860fa99cf3 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.352s user 0.046s sys 0.008s
I20240501 20:10:20.959722 10240 ts_tablet_manager.cc:1888] T 8a81cfcb50fb442a834d6f9de70bb0ca P db0742ee3860468590eefef6ae6074fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:20.959913 7287 raft_consensus.cc:2932] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:20.962172 10164 raft_consensus.cc:348] T dfaa3794205d482c90aa4f860fa99cf3 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:20.962443 10164 raft_consensus.cc:729] T dfaa3794205d482c90aa4f860fa99cf3 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: LEARNER
I20240501 20:10:20.962680 10164 consensus_queue.cc:260] T dfaa3794205d482c90aa4f860fa99cf3 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:20.964576 10164 ts_tablet_manager.cc:1398] T dfaa3794205d482c90aa4f860fa99cf3 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 20:10:20.965236 4254 tablet_copy_service.cc:342] P dca57823003e4729b769ca12367d7530: Request end of tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-dfaa3794205d482c90aa4f860fa99cf3 received from {username='slave'} at 127.29.219.69:51289
I20240501 20:10:20.965355 4254 tablet_copy_service.cc:433] P dca57823003e4729b769ca12367d7530: ending tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-dfaa3794205d482c90aa4f860fa99cf3 on tablet dfaa3794205d482c90aa4f860fa99cf3 with peer ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:20.966414 4087 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:20.971454 10239 ts_tablet_manager.cc:1888] T 8a81cfcb50fb442a834d6f9de70bb0ca P ef0a4ada548446b3a531cb39a3c4e38d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:20.971589 10239 log.cc:1196] T 8a81cfcb50fb442a834d6f9de70bb0ca P ef0a4ada548446b3a531cb39a3c4e38d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/wals/8a81cfcb50fb442a834d6f9de70bb0ca
I20240501 20:10:20.972169 10239 ts_tablet_manager.cc:1909] T 8a81cfcb50fb442a834d6f9de70bb0ca P ef0a4ada548446b3a531cb39a3c4e38d: Deleting consensus metadata
I20240501 20:10:20.973416 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet 8a81cfcb50fb442a834d6f9de70bb0ca (table rebalancer_dynamic_table_new_name [id=ba157bafcad84545ad579f1f1d68ba4a]) successfully deleted
I20240501 20:10:20.985533 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 0e85e0affad146d8a7c6b6751012dc70 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=756dbeb8af6e453793147c0af8465921]), partition=
I20240501 20:10:20.985854 7400 tablet_service.cc:1467] Processing CreateTablet for tablet 0e85e0affad146d8a7c6b6751012dc70 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=756dbeb8af6e453793147c0af8465921]), partition=
I20240501 20:10:20.985917 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e85e0affad146d8a7c6b6751012dc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:20.986205 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e85e0affad146d8a7c6b6751012dc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:20.987676 4089 catalog_manager.cc:5484] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index 6 to 7, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:20.990270 7133 tablet_service.cc:1467] Processing CreateTablet for tablet 0e85e0affad146d8a7c6b6751012dc70 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=756dbeb8af6e453793147c0af8465921]), partition=
I20240501 20:10:20.999148 7133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e85e0affad146d8a7c6b6751012dc70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:21.011175 10240 log.cc:1196] T 8a81cfcb50fb442a834d6f9de70bb0ca P db0742ee3860468590eefef6ae6074fd: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/wals/8a81cfcb50fb442a834d6f9de70bb0ca
I20240501 20:10:21.011940 10240 ts_tablet_manager.cc:1909] T 8a81cfcb50fb442a834d6f9de70bb0ca P db0742ee3860468590eefef6ae6074fd: Deleting consensus metadata
I20240501 20:10:21.014842 4076 catalog_manager.cc:4830] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): tablet 8a81cfcb50fb442a834d6f9de70bb0ca (table rebalancer_dynamic_table_new_name [id=ba157bafcad84545ad579f1f1d68ba4a]) successfully deleted
I20240501 20:10:21.023567 10254 tablet_bootstrap.cc:492] T 0e85e0affad146d8a7c6b6751012dc70 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:21.029446 8785 consensus_queue.cc:1035] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:21.035871 4631 raft_consensus.cc:1203] T 129322eab07149d2a6fb7eef96100e7a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:21.039459 10254 tablet_bootstrap.cc:654] T 0e85e0affad146d8a7c6b6751012dc70 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:21.048099 10254 tablet_bootstrap.cc:492] T 0e85e0affad146d8a7c6b6751012dc70 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:21.053694 10254 ts_tablet_manager.cc:1367] T 0e85e0affad146d8a7c6b6751012dc70 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.000s
I20240501 20:10:21.055338 10254 raft_consensus.cc:348] T 0e85e0affad146d8a7c6b6751012dc70 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:21.056449 10254 raft_consensus.cc:374] T 0e85e0affad146d8a7c6b6751012dc70 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:21.056710 10254 raft_consensus.cc:729] T 0e85e0affad146d8a7c6b6751012dc70 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:21.057161 10254 consensus_queue.cc:260] T 0e85e0affad146d8a7c6b6751012dc70 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:21.059620 10254 ts_tablet_manager.cc:1398] T 0e85e0affad146d8a7c6b6751012dc70 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:10:21.063517 10255 tablet_bootstrap.cc:492] T 0e85e0affad146d8a7c6b6751012dc70 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:21.063887 10256 tablet_bootstrap.cc:492] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:21.075577 9744 consensus_queue.cc:1035] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:21.081005 10256 tablet_bootstrap.cc:654] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:21.085381 10255 tablet_bootstrap.cc:654] T 0e85e0affad146d8a7c6b6751012dc70 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:21.088666 9802 consensus_queue.cc:1035] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:21.092851 4363 raft_consensus.cc:1203] T c1e7ac38dc9c44d29e2d8127904407eb P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:21.094282 10256 tablet_bootstrap.cc:492] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:21.094444 10256 ts_tablet_manager.cc:1367] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.011s
I20240501 20:10:21.095317 10256 raft_consensus.cc:348] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:21.095535 10256 raft_consensus.cc:374] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:21.095638 10256 raft_consensus.cc:729] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:21.095904 10256 consensus_queue.cc:260] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:21.096529 10256 ts_tablet_manager.cc:1398] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:10:21.109540 10190 ts_tablet_manager.cc:902] T 9e31f8d1809244efae05ab9254c163ce P 1d34e18a1d394d3f91c9f50d1c7bea4e: Initiating tablet copy from peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:21.149636 4496 raft_consensus.cc:1203] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:21.154438 9926 raft_consensus.cc:482] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:21.154595 9926 raft_consensus.cc:504] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:21.155465 9926 leader_election.cc:288] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:21.164535 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e85e0affad146d8a7c6b6751012dc70" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:21.164801 4631 raft_consensus.cc:2454] T 0e85e0affad146d8a7c6b6751012dc70 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:21.164983 7416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e85e0affad146d8a7c6b6751012dc70" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
W20240501 20:10:21.165850 7091 leader_election.cc:341] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:21.166134 7091 leader_election.cc:302] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, db0742ee3860468590eefef6ae6074fd; no voters: b9957d4086bc4a76b18ed9265a5739d5
I20240501 20:10:21.166631 10252 raft_consensus.cc:2781] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:21.166723 10252 raft_consensus.cc:482] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:21.166839 10252 raft_consensus.cc:3037] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:21.169854 10252 raft_consensus.cc:504] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:21.170475 10252 leader_election.cc:288] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:21.172446 7022 raft_consensus.cc:1203] T dfaa3794205d482c90aa4f860fa99cf3 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:10:21.176631 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e85e0affad146d8a7c6b6751012dc70" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:10:21.176842 4631 raft_consensus.cc:3037] T 0e85e0affad146d8a7c6b6751012dc70 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:21.179836 4631 raft_consensus.cc:2454] T 0e85e0affad146d8a7c6b6751012dc70 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:21.180759 7091 leader_election.cc:302] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:21.181500 9926 raft_consensus.cc:2781] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:21.181672 9926 raft_consensus.cc:686] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Becoming Leader. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Running, Role: LEADER
I20240501 20:10:21.181947 9926 consensus_queue.cc:237] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:21.194598 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e85e0affad146d8a7c6b6751012dc70" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
W20240501 20:10:21.196285 7091 leader_election.cc:341] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:10:21.223418 4564 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.65:42941, user_credentials={real_user=slave}} blocked reactor thread for 86219.1us
I20240501 20:10:21.225509 4089 catalog_manager.cc:5484] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd reported cstate change: term changed from 0 to 1, leader changed from <none> to db0742ee3860468590eefef6ae6074fd (127.29.219.70). New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } }
W20240501 20:10:21.232093 7090 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.69:46237, user_credentials={real_user=slave}} blocked reactor thread for 48199.7us
I20240501 20:10:21.244194 10228 consensus_queue.cc:1035] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:21.253060 10253 consensus_queue.cc:1035] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:21.256736 10202 ts_tablet_manager.cc:902] T 61af956bc72b4e3e994a7daf58d1a699 P 10b47782558c41ee91f5e164a372049a: Initiating tablet copy from peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:21.268867 10228 consensus_queue.cc:1035] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:21.296334 10202 tablet_copy_client.cc:308] T 61af956bc72b4e3e994a7daf58d1a699 P 10b47782558c41ee91f5e164a372049a: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.67:34595
I20240501 20:10:21.296988 4519 tablet_copy_service.cc:140] P 3e974c03d3c546f9ab6c91e762a5426b: Received BeginTabletCopySession request for tablet 61af956bc72b4e3e994a7daf58d1a699 from peer 10b47782558c41ee91f5e164a372049a ({username='slave'} at 127.29.219.66:47817)
I20240501 20:10:21.297120 4519 tablet_copy_service.cc:161] P 3e974c03d3c546f9ab6c91e762a5426b: Beginning new tablet copy session on tablet 61af956bc72b4e3e994a7daf58d1a699 from peer 10b47782558c41ee91f5e164a372049a at {username='slave'} at 127.29.219.66:47817: session id = 10b47782558c41ee91f5e164a372049a-61af956bc72b4e3e994a7daf58d1a699
I20240501 20:10:21.300019 4519 tablet_copy_source_session.cc:205] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:21.302929 10202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61af956bc72b4e3e994a7daf58d1a699. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:21.309334 10202 tablet_copy_client.cc:783] T 61af956bc72b4e3e994a7daf58d1a699 P 10b47782558c41ee91f5e164a372049a: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:21.309609 10202 tablet_copy_client.cc:647] T 61af956bc72b4e3e994a7daf58d1a699 P 10b47782558c41ee91f5e164a372049a: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:21.312287 10202 tablet_copy_client.cc:523] T 61af956bc72b4e3e994a7daf58d1a699 P 10b47782558c41ee91f5e164a372049a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:21.316661 10202 tablet_bootstrap.cc:492] T 61af956bc72b4e3e994a7daf58d1a699 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:21.318601 10255 tablet_bootstrap.cc:492] T 0e85e0affad146d8a7c6b6751012dc70 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:21.318878 10255 ts_tablet_manager.cc:1367] T 0e85e0affad146d8a7c6b6751012dc70 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.256s user 0.029s sys 0.070s
I20240501 20:10:21.319981 10255 raft_consensus.cc:348] T 0e85e0affad146d8a7c6b6751012dc70 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:21.320469 10245 tablet_copy_client.cc:308] T 5a7b2f01b5684df5800987591a390a17 P dca57823003e4729b769ca12367d7530: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.66:40689
I20240501 20:10:21.321171 10255 raft_consensus.cc:374] T 0e85e0affad146d8a7c6b6751012dc70 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:21.321202 4388 tablet_copy_service.cc:140] P 10b47782558c41ee91f5e164a372049a: Received BeginTabletCopySession request for tablet 5a7b2f01b5684df5800987591a390a17 from peer dca57823003e4729b769ca12367d7530 ({username='slave'} at 127.29.219.65:41491)
I20240501 20:10:21.321347 4388 tablet_copy_service.cc:161] P 10b47782558c41ee91f5e164a372049a: Beginning new tablet copy session on tablet 5a7b2f01b5684df5800987591a390a17 from peer dca57823003e4729b769ca12367d7530 at {username='slave'} at 127.29.219.65:41491: session id = dca57823003e4729b769ca12367d7530-5a7b2f01b5684df5800987591a390a17
I20240501 20:10:21.321395 10255 raft_consensus.cc:729] T 0e85e0affad146d8a7c6b6751012dc70 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:21.321792 10255 consensus_queue.cc:260] T 0e85e0affad146d8a7c6b6751012dc70 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:21.322453 10255 ts_tablet_manager.cc:1398] T 0e85e0affad146d8a7c6b6751012dc70 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:21.324975 4388 tablet_copy_source_session.cc:205] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:21.326371 10245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a7b2f01b5684df5800987591a390a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:21.334206 4089 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:21.335980 10245 tablet_copy_client.cc:783] T 5a7b2f01b5684df5800987591a390a17 P dca57823003e4729b769ca12367d7530: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:21.336365 10245 tablet_copy_client.cc:647] T 5a7b2f01b5684df5800987591a390a17 P dca57823003e4729b769ca12367d7530: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:21.348887 10202 tablet_bootstrap.cc:492] T 61af956bc72b4e3e994a7daf58d1a699 P 10b47782558c41ee91f5e164a372049a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:21.349303 10202 tablet_bootstrap.cc:492] T 61af956bc72b4e3e994a7daf58d1a699 P 10b47782558c41ee91f5e164a372049a: Bootstrap complete.
I20240501 20:10:21.349574 10202 ts_tablet_manager.cc:1367] T 61af956bc72b4e3e994a7daf58d1a699 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.033s user 0.014s sys 0.004s
I20240501 20:10:21.349965 7419 raft_consensus.cc:3037] T 0e85e0affad146d8a7c6b6751012dc70 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:21.350375 10202 raft_consensus.cc:348] T 61af956bc72b4e3e994a7daf58d1a699 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:21.350631 10202 raft_consensus.cc:729] T 61af956bc72b4e3e994a7daf58d1a699 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: LEARNER
I20240501 20:10:21.350841 10202 consensus_queue.cc:260] T 61af956bc72b4e3e994a7daf58d1a699 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:21.351367 10202 ts_tablet_manager.cc:1398] T 61af956bc72b4e3e994a7daf58d1a699 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20240501 20:10:21.352051 4519 tablet_copy_service.cc:342] P 3e974c03d3c546f9ab6c91e762a5426b: Request end of tablet copy session 10b47782558c41ee91f5e164a372049a-61af956bc72b4e3e994a7daf58d1a699 received from {username='slave'} at 127.29.219.66:47817
I20240501 20:10:21.352177 4519 tablet_copy_service.cc:433] P 3e974c03d3c546f9ab6c91e762a5426b: ending tablet copy session 10b47782558c41ee91f5e164a372049a-61af956bc72b4e3e994a7daf58d1a699 on tablet 61af956bc72b4e3e994a7daf58d1a699 with peer 10b47782558c41ee91f5e164a372049a
I20240501 20:10:21.352916 7419 raft_consensus.cc:1261] T 0e85e0affad146d8a7c6b6751012dc70 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:21.354583 10252 consensus_queue.cc:1035] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:21.362174 10245 tablet_copy_client.cc:523] T 5a7b2f01b5684df5800987591a390a17 P dca57823003e4729b769ca12367d7530: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:21.363617 4089 catalog_manager.cc:5576] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817) does not have the latest schema for tablet 0e85e0affad146d8a7c6b6751012dc70 (table rebalancer_dynamic_table [id=756dbeb8af6e453793147c0af8465921]). Expected version 1 got 0
I20240501 20:10:21.366264 10245 tablet_bootstrap.cc:492] T 5a7b2f01b5684df5800987591a390a17 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:21.378214 10293 ts_tablet_manager.cc:902] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd: Initiating tablet copy from peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:21.379911 10296 tablet.cc:1722] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:21.380625 10296 tablet.cc:1620] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:10:21.385308 10297 tablet.cc:1722] T 0e85e0affad146d8a7c6b6751012dc70 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:21.385934 10297 tablet.cc:1620] T 0e85e0affad146d8a7c6b6751012dc70 P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:21.386157 9953 raft_consensus.cc:1050] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530: attempting to promote NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b to VOTER
I20240501 20:10:21.387212 9953 consensus_queue.cc:237] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:21.389539 10190 tablet_copy_client.cc:308] T 9e31f8d1809244efae05ab9254c163ce P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.72:39817
I20240501 20:10:21.390360 7441 tablet_copy_service.cc:140] P b9957d4086bc4a76b18ed9265a5739d5: Received BeginTabletCopySession request for tablet 9e31f8d1809244efae05ab9254c163ce from peer 1d34e18a1d394d3f91c9f50d1c7bea4e ({username='slave'} at 127.29.219.68:34551)
I20240501 20:10:21.390547 7441 tablet_copy_service.cc:161] P b9957d4086bc4a76b18ed9265a5739d5: Beginning new tablet copy session on tablet 9e31f8d1809244efae05ab9254c163ce from peer 1d34e18a1d394d3f91c9f50d1c7bea4e at {username='slave'} at 127.29.219.68:34551: session id = 1d34e18a1d394d3f91c9f50d1c7bea4e-9e31f8d1809244efae05ab9254c163ce
I20240501 20:10:21.392661 10245 tablet_bootstrap.cc:492] T 5a7b2f01b5684df5800987591a390a17 P dca57823003e4729b769ca12367d7530: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:21.393038 10245 tablet_bootstrap.cc:492] T 5a7b2f01b5684df5800987591a390a17 P dca57823003e4729b769ca12367d7530: Bootstrap complete.
I20240501 20:10:21.393317 10245 ts_tablet_manager.cc:1367] T 5a7b2f01b5684df5800987591a390a17 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.000s
I20240501 20:10:21.393482 7441 tablet_copy_source_session.cc:205] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:21.393496 4088 catalog_manager.cc:5576] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817) does not have the latest schema for tablet 0e85e0affad146d8a7c6b6751012dc70 (table rebalancer_dynamic_table [id=756dbeb8af6e453793147c0af8465921]). Expected version 1 got 0
I20240501 20:10:21.394094 10245 raft_consensus.cc:348] T 5a7b2f01b5684df5800987591a390a17 P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:21.394357 10245 raft_consensus.cc:729] T 5a7b2f01b5684df5800987591a390a17 P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: LEARNER
I20240501 20:10:21.394582 4627 raft_consensus.cc:1261] T 0e85e0affad146d8a7c6b6751012dc70 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:21.394554 10245 consensus_queue.cc:260] T 5a7b2f01b5684df5800987591a390a17 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:21.395296 10245 ts_tablet_manager.cc:1398] T 5a7b2f01b5684df5800987591a390a17 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20240501 20:10:21.397475 10277 consensus_queue.cc:1035] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:21.400705 4089 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=756dbeb8af6e453793147c0af8465921] alter complete (version 1)
I20240501 20:10:21.415616 4233 raft_consensus.cc:1203] T 5a7b2f01b5684df5800987591a390a17 P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:21.416657 4388 tablet_copy_service.cc:342] P 10b47782558c41ee91f5e164a372049a: Request end of tablet copy session dca57823003e4729b769ca12367d7530-5a7b2f01b5684df5800987591a390a17 received from {username='slave'} at 127.29.219.65:41491
I20240501 20:10:21.416818 4388 tablet_copy_service.cc:433] P 10b47782558c41ee91f5e164a372049a: ending tablet copy session dca57823003e4729b769ca12367d7530-5a7b2f01b5684df5800987591a390a17 on tablet 5a7b2f01b5684df5800987591a390a17 with peer dca57823003e4729b769ca12367d7530
I20240501 20:10:21.421288 7285 consensus_queue.cc:237] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:21.425743 10190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e31f8d1809244efae05ab9254c163ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:21.434793 7022 raft_consensus.cc:1261] T e7f35eafb0be4f4eab73133917b22bbf P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:21.435612 7222 consensus_peers.cc:485] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7f35eafb0be4f4eab73133917b22bbf. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:21.435580 10116 consensus_queue.cc:1035] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:21.439214 9956 raft_consensus.cc:2932] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:21.440958 4628 raft_consensus.cc:1261] T e7f35eafb0be4f4eab73133917b22bbf P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:21.441583 10116 consensus_queue.cc:1035] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:21.447748 7022 raft_consensus.cc:2932] T e7f35eafb0be4f4eab73133917b22bbf P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:21.451016 4087 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:21.455072 4092 catalog_manager.cc:5484] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:21.478825 4630 raft_consensus.cc:2932] T e7f35eafb0be4f4eab73133917b22bbf P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:21.479331 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 3bdd6de7ac584032aa309269bd0770cf (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=0a6696a672f449e89bcc33228cf8c494]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:21.479557 7002 tablet_service.cc:1467] Processing CreateTablet for tablet 3bdd6de7ac584032aa309269bd0770cf (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=0a6696a672f449e89bcc33228cf8c494]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:21.479749 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bdd6de7ac584032aa309269bd0770cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:21.480067 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bdd6de7ac584032aa309269bd0770cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:21.482456 7419 raft_consensus.cc:1261] T 264d3fcd3fae4acd8d76306c0f32ed72 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:21.482522 7155 raft_consensus.cc:1261] T 264d3fcd3fae4acd8d76306c0f32ed72 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:21.483285 9802 consensus_queue.cc:1035] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:21.483762 4496 raft_consensus.cc:1261] T 264d3fcd3fae4acd8d76306c0f32ed72 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:21.483803 9802 consensus_queue.cc:1035] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:21.484465 9802 consensus_queue.cc:1035] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:21.486387 7133 tablet_service.cc:1467] Processing CreateTablet for tablet 3bdd6de7ac584032aa309269bd0770cf (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=0a6696a672f449e89bcc33228cf8c494]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:21.486945 7133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bdd6de7ac584032aa309269bd0770cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:21.495468 10308 tablet_bootstrap.cc:492] T 3bdd6de7ac584032aa309269bd0770cf P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:21.497468 10308 tablet_bootstrap.cc:654] T 3bdd6de7ac584032aa309269bd0770cf P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:21.504663 7400 tablet_service.cc:1467] Processing CreateTablet for tablet 9709a8bb107742b79078f91da9fab8f6 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=0a6696a672f449e89bcc33228cf8c494]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:21.505254 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9709a8bb107742b79078f91da9fab8f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:21.506421 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 9709a8bb107742b79078f91da9fab8f6 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=0a6696a672f449e89bcc33228cf8c494]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:21.506971 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9709a8bb107742b79078f91da9fab8f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:21.507197 7134 tablet_service.cc:1467] Processing CreateTablet for tablet 9709a8bb107742b79078f91da9fab8f6 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=0a6696a672f449e89bcc33228cf8c494]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:21.507697 7134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9709a8bb107742b79078f91da9fab8f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:21.511122 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 376ae6034f394a0fb0cab0d8a9241134 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=0a6696a672f449e89bcc33228cf8c494]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:21.511356 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 376ae6034f394a0fb0cab0d8a9241134 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=0a6696a672f449e89bcc33228cf8c494]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:21.511561 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 376ae6034f394a0fb0cab0d8a9241134. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:21.511824 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 376ae6034f394a0fb0cab0d8a9241134. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:21.517058 10256 tablet_bootstrap.cc:492] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:21.518673 10256 tablet_bootstrap.cc:654] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:21.523988 10254 tablet_bootstrap.cc:492] T 9709a8bb107742b79078f91da9fab8f6 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:21.525301 10254 tablet_bootstrap.cc:654] T 9709a8bb107742b79078f91da9fab8f6 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:21.525851 7267 tablet_service.cc:1467] Processing CreateTablet for tablet 376ae6034f394a0fb0cab0d8a9241134 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=0a6696a672f449e89bcc33228cf8c494]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:21.526304 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 376ae6034f394a0fb0cab0d8a9241134. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:21.530721 4093 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:21.549387 10254 tablet_bootstrap.cc:492] T 9709a8bb107742b79078f91da9fab8f6 P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:21.550107 10296 tablet.cc:1722] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:21.552867 10297 tablet.cc:1722] T 0e85e0affad146d8a7c6b6751012dc70 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:21.553442 10297 tablet.cc:1620] T 0e85e0affad146d8a7c6b6751012dc70 P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:21.553983 10296 tablet.cc:1620] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:10:21.558358 10317 tablet_bootstrap.cc:492] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:21.560566 10317 tablet_bootstrap.cc:654] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:21.566005 10256 tablet_bootstrap.cc:492] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:21.566219 10256 ts_tablet_manager.cc:1367] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.009s
I20240501 20:10:21.567051 10308 tablet_bootstrap.cc:492] T 3bdd6de7ac584032aa309269bd0770cf P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:21.567207 10308 ts_tablet_manager.cc:1367] T 3bdd6de7ac584032aa309269bd0770cf P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.072s user 0.002s sys 0.004s
I20240501 20:10:21.567818 4092 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=756dbeb8af6e453793147c0af8465921] alter complete (version 2)
I20240501 20:10:21.568023 10308 raft_consensus.cc:348] T 3bdd6de7ac584032aa309269bd0770cf P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:21.556547 10190 tablet_copy_client.cc:783] T 9e31f8d1809244efae05ab9254c163ce P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:21.568557 10254 ts_tablet_manager.cc:1367] T 9709a8bb107742b79078f91da9fab8f6 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.010s
I20240501 20:10:21.568637 10190 tablet_copy_client.cc:647] T 9e31f8d1809244efae05ab9254c163ce P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:21.570515 10190 tablet_copy_client.cc:523] T 9e31f8d1809244efae05ab9254c163ce P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:21.574945 10190 tablet_bootstrap.cc:492] T 9e31f8d1809244efae05ab9254c163ce P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:21.580534 10293 tablet_copy_client.cc:308] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.66:40689
I20240501 20:10:21.581511 4388 tablet_copy_service.cc:140] P 10b47782558c41ee91f5e164a372049a: Received BeginTabletCopySession request for tablet c66514deec5744da832bc030a582c46a from peer db0742ee3860468590eefef6ae6074fd ({username='slave'} at 127.29.219.70:42453)
I20240501 20:10:21.581709 4388 tablet_copy_service.cc:161] P 10b47782558c41ee91f5e164a372049a: Beginning new tablet copy session on tablet c66514deec5744da832bc030a582c46a from peer db0742ee3860468590eefef6ae6074fd at {username='slave'} at 127.29.219.70:42453: session id = db0742ee3860468590eefef6ae6074fd-c66514deec5744da832bc030a582c46a
I20240501 20:10:21.581753 10256 raft_consensus.cc:348] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:21.582005 10256 raft_consensus.cc:374] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:21.582095 10256 raft_consensus.cc:729] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:21.582335 10256 consensus_queue.cc:260] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:21.575892 10254 raft_consensus.cc:348] T 9709a8bb107742b79078f91da9fab8f6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:21.582602 10254 raft_consensus.cc:374] T 9709a8bb107742b79078f91da9fab8f6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:21.582715 10254 raft_consensus.cc:729] T 9709a8bb107742b79078f91da9fab8f6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:21.582966 10254 consensus_queue.cc:260] T 9709a8bb107742b79078f91da9fab8f6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:21.583235 10308 raft_consensus.cc:374] T 3bdd6de7ac584032aa309269bd0770cf P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:21.583319 10308 raft_consensus.cc:729] T 3bdd6de7ac584032aa309269bd0770cf P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:21.583511 10308 consensus_queue.cc:260] T 3bdd6de7ac584032aa309269bd0770cf P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:21.584300 10254 ts_tablet_manager.cc:1398] T 9709a8bb107742b79078f91da9fab8f6 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240501 20:10:21.584954 4388 tablet_copy_source_session.cc:205] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:21.586890 10293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c66514deec5744da832bc030a582c46a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:21.590857 10075 raft_consensus.cc:1050] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97: attempting to promote NON_VOTER 10b47782558c41ee91f5e164a372049a to VOTER
I20240501 20:10:21.591771 10308 ts_tablet_manager.cc:1398] T 3bdd6de7ac584032aa309269bd0770cf P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240501 20:10:21.592151 10308 tablet_bootstrap.cc:492] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:21.594043 10308 tablet_bootstrap.cc:654] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:21.595129 10256 ts_tablet_manager.cc:1398] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.029s user 0.000s sys 0.003s
I20240501 20:10:21.595250 10075 consensus_queue.cc:237] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } }
I20240501 20:10:21.606838 10256 tablet_bootstrap.cc:492] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:21.609206 10256 tablet_bootstrap.cc:654] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:21.610755 10255 tablet_bootstrap.cc:492] T 376ae6034f394a0fb0cab0d8a9241134 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:21.612859 10255 tablet_bootstrap.cc:654] T 376ae6034f394a0fb0cab0d8a9241134 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:21.623992 10275 raft_consensus.cc:482] T 9709a8bb107742b79078f91da9fab8f6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:21.624204 10275 raft_consensus.cc:504] T 9709a8bb107742b79078f91da9fab8f6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:21.625097 10275 leader_election.cc:288] T 9709a8bb107742b79078f91da9fab8f6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:21.626068 7154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9709a8bb107742b79078f91da9fab8f6" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:21.627524 7416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9709a8bb107742b79078f91da9fab8f6" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
W20240501 20:10:21.628360 4562 leader_election.cc:341] T 9709a8bb107742b79078f91da9fab8f6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:10:21.628842 4565 leader_election.cc:341] T 9709a8bb107742b79078f91da9fab8f6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:21.628976 4565 leader_election.cc:302] T 9709a8bb107742b79078f91da9fab8f6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e; no voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd
I20240501 20:10:21.629557 10275 raft_consensus.cc:2726] T 9709a8bb107742b79078f91da9fab8f6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:21.631978 10313 tablet.cc:1722] T 0e85e0affad146d8a7c6b6751012dc70 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:21.632573 10313 tablet.cc:1620] T 0e85e0affad146d8a7c6b6751012dc70 P 1d34e18a1d394d3f91c9f50d1c7bea4e: MemRowSet was empty: no flush needed.
I20240501 20:10:21.633669 10308 tablet_bootstrap.cc:492] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:21.634042 10308 ts_tablet_manager.cc:1367] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20240501 20:10:21.636030 10308 raft_consensus.cc:348] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:21.637197 10221 raft_consensus.cc:1050] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5: attempting to promote NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e to VOTER
I20240501 20:10:21.637615 4363 raft_consensus.cc:1261] T c1e7ac38dc9c44d29e2d8127904407eb P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:21.638197 9956 consensus_queue.cc:1035] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:21.638259 10221 consensus_queue.cc:237] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } }
I20240501 20:10:21.647739 7154 raft_consensus.cc:1261] T c1e7ac38dc9c44d29e2d8127904407eb P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:21.648367 10075 consensus_queue.cc:1035] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:21.649298 7419 raft_consensus.cc:2932] T 264d3fcd3fae4acd8d76306c0f32ed72 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:21.650784 10314 tablet_bootstrap.cc:492] T 3bdd6de7ac584032aa309269bd0770cf P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:21.652649 4498 raft_consensus.cc:2932] T 264d3fcd3fae4acd8d76306c0f32ed72 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:21.654939 4092 catalog_manager.cc:5576] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205) does not have the latest schema for tablet 0e85e0affad146d8a7c6b6751012dc70 (table rebalancer_dynamic_table [id=756dbeb8af6e453793147c0af8465921]). Expected version 2 got 0
I20240501 20:10:21.654994 10190 tablet_bootstrap.cc:492] T 9e31f8d1809244efae05ab9254c163ce P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:21.659075 10308 raft_consensus.cc:374] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:21.659273 10308 raft_consensus.cc:729] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:21.659590 10308 consensus_queue.cc:260] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:21.659797 10190 tablet_bootstrap.cc:492] T 9e31f8d1809244efae05ab9254c163ce P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap complete.
I20240501 20:10:21.646703 8785 raft_consensus.cc:2932] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:21.660115 10190 ts_tablet_manager.cc:1367] T 9e31f8d1809244efae05ab9254c163ce P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.085s user 0.008s sys 0.000s
I20240501 20:10:21.660846 10190 raft_consensus.cc:348] T 9e31f8d1809244efae05ab9254c163ce P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:21.661067 10190 raft_consensus.cc:729] T 9e31f8d1809244efae05ab9254c163ce P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: LEARNER
I20240501 20:10:21.661243 10190 consensus_queue.cc:260] T 9e31f8d1809244efae05ab9254c163ce P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:21.661815 10190 ts_tablet_manager.cc:1398] T 9e31f8d1809244efae05ab9254c163ce P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:21.662163 4087 catalog_manager.cc:5576] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205) does not have the latest schema for tablet 0e85e0affad146d8a7c6b6751012dc70 (table rebalancer_dynamic_table [id=756dbeb8af6e453793147c0af8465921]). Expected version 2 got 1
I20240501 20:10:21.663102 10308 ts_tablet_manager.cc:1398] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.029s user 0.000s sys 0.002s
I20240501 20:10:21.658195 7416 raft_consensus.cc:1261] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:21.662431 7441 tablet_copy_service.cc:342] P b9957d4086bc4a76b18ed9265a5739d5: Request end of tablet copy session 1d34e18a1d394d3f91c9f50d1c7bea4e-9e31f8d1809244efae05ab9254c163ce received from {username='slave'} at 127.29.219.68:34551
I20240501 20:10:21.663497 7441 tablet_copy_service.cc:433] P b9957d4086bc4a76b18ed9265a5739d5: ending tablet copy session 1d34e18a1d394d3f91c9f50d1c7bea4e-9e31f8d1809244efae05ab9254c163ce on tablet 9e31f8d1809244efae05ab9254c163ce with peer 1d34e18a1d394d3f91c9f50d1c7bea4e
I20240501 20:10:21.664341 10255 tablet_bootstrap.cc:492] T 376ae6034f394a0fb0cab0d8a9241134 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:21.669901 10314 tablet_bootstrap.cc:654] T 3bdd6de7ac584032aa309269bd0770cf P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:21.673923 9956 consensus_queue.cc:1035] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:21.674062 10255 ts_tablet_manager.cc:1367] T 376ae6034f394a0fb0cab0d8a9241134 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.054s user 0.009s sys 0.002s
I20240501 20:10:21.675485 10255 raft_consensus.cc:348] T 376ae6034f394a0fb0cab0d8a9241134 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:21.675701 10255 raft_consensus.cc:374] T 376ae6034f394a0fb0cab0d8a9241134 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:21.675774 10255 raft_consensus.cc:729] T 376ae6034f394a0fb0cab0d8a9241134 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:21.675997 10255 consensus_queue.cc:260] T 376ae6034f394a0fb0cab0d8a9241134 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:21.678880 4092 catalog_manager.cc:5484] T 264d3fcd3fae4acd8d76306c0f32ed72 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:21.684087 7287 raft_consensus.cc:1261] T 129322eab07149d2a6fb7eef96100e7a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:21.680610 10145 raft_consensus.cc:1050] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d: attempting to promote NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b to VOTER
I20240501 20:10:21.686851 10145 consensus_queue.cc:237] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:21.690469 10293 tablet_copy_client.cc:783] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:21.688427 7152 raft_consensus.cc:2932] T 264d3fcd3fae4acd8d76306c0f32ed72 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:21.691231 4087 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:21.705961 10314 tablet_bootstrap.cc:492] T 3bdd6de7ac584032aa309269bd0770cf P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:21.707253 10314 ts_tablet_manager.cc:1367] T 3bdd6de7ac584032aa309269bd0770cf P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.057s user 0.004s sys 0.000s
I20240501 20:10:21.708346 9802 raft_consensus.cc:1050] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530: attempting to promote NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d to VOTER
I20240501 20:10:21.709089 4093 catalog_manager.cc:5576] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205) does not have the latest schema for tablet 0e85e0affad146d8a7c6b6751012dc70 (table rebalancer_dynamic_table_new_name [id=756dbeb8af6e453793147c0af8465921]). Expected version 3 got 1
I20240501 20:10:21.709383 9802 consensus_queue.cc:237] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:21.710029 10314 raft_consensus.cc:348] T 3bdd6de7ac584032aa309269bd0770cf P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:21.711573 10314 raft_consensus.cc:374] T 3bdd6de7ac584032aa309269bd0770cf P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:21.712164 10317 tablet_bootstrap.cc:492] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:21.712368 10314 raft_consensus.cc:729] T 3bdd6de7ac584032aa309269bd0770cf P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:21.712651 10296 tablet.cc:1722] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:21.712489 10317 ts_tablet_manager.cc:1367] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.154s user 0.011s sys 0.015s
I20240501 20:10:21.715744 10317 raft_consensus.cc:348] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:21.716071 10317 raft_consensus.cc:374] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:21.716202 10317 raft_consensus.cc:729] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:21.716619 10317 consensus_queue.cc:260] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:21.717366 10314 consensus_queue.cc:260] T 3bdd6de7ac584032aa309269bd0770cf P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:21.717396 9956 raft_consensus.cc:2932] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:21.721976 10293 tablet_copy_client.cc:647] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:21.723793 10293 tablet_copy_client.cc:523] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:21.725445 7155 raft_consensus.cc:2932] T c1e7ac38dc9c44d29e2d8127904407eb P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:21.717415 10313 tablet.cc:1722] T 0e85e0affad146d8a7c6b6751012dc70 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:21.738389 10255 ts_tablet_manager.cc:1398] T 376ae6034f394a0fb0cab0d8a9241134 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.064s user 0.001s sys 0.001s
I20240501 20:10:21.740065 4093 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table_new_name [id=756dbeb8af6e453793147c0af8465921] alter complete (version 3)
I20240501 20:10:21.740193 4631 raft_consensus.cc:1261] T 129322eab07149d2a6fb7eef96100e7a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:21.740257 10224 consensus_queue.cc:1035] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20240501 20:10:21.736572 4363 raft_consensus.cc:2932] T c1e7ac38dc9c44d29e2d8127904407eb P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:21.740803 10046 consensus_queue.cc:1035] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:21.742385 10255 tablet_bootstrap.cc:492] T 9709a8bb107742b79078f91da9fab8f6 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:21.746429 10255 tablet_bootstrap.cc:654] T 9709a8bb107742b79078f91da9fab8f6 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:21.749578 10293 tablet_bootstrap.cc:492] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:21.750169 4093 catalog_manager.cc:5484] T c1e7ac38dc9c44d29e2d8127904407eb P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:21.751888 8785 raft_consensus.cc:482] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:21.752043 8785 raft_consensus.cc:504] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:21.752213 4234 consensus_queue.cc:237] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:21.758652 7417 raft_consensus.cc:1261] T 264d3fcd3fae4acd8d76306c0f32ed72 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:21.759331 10297 tablet.cc:1722] T 0e85e0affad146d8a7c6b6751012dc70 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:21.760831 4496 raft_consensus.cc:1261] T 264d3fcd3fae4acd8d76306c0f32ed72 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:21.760985 8785 leader_election.cc:288] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:21.761010 10075 raft_consensus.cc:482] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:21.761188 10075 raft_consensus.cc:504] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:21.761394 8785 consensus_queue.cc:1035] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:21.762378 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "376ae6034f394a0fb0cab0d8a9241134" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:21.762594 7285 raft_consensus.cc:2454] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:21.763039 4168 leader_election.cc:302] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:21.763408 7417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "376ae6034f394a0fb0cab0d8a9241134" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:21.763621 7417 raft_consensus.cc:2454] T 376ae6034f394a0fb0cab0d8a9241134 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:21.763954 7223 leader_election.cc:302] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:21.765486 9956 raft_consensus.cc:2781] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:21.765596 9956 raft_consensus.cc:482] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:21.765724 9956 raft_consensus.cc:3037] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:21.766539 8785 raft_consensus.cc:2781] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:21.766665 8785 raft_consensus.cc:482] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:21.766738 8785 raft_consensus.cc:3037] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:21.767184 10256 tablet_bootstrap.cc:492] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:21.767346 10256 ts_tablet_manager.cc:1367] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.161s user 0.008s sys 0.007s
I20240501 20:10:21.768477 9956 raft_consensus.cc:504] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:21.769433 8785 raft_consensus.cc:504] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:21.770601 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "376ae6034f394a0fb0cab0d8a9241134" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:10:21.770843 4233 raft_consensus.cc:2379] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bceeae1c79964acabdae8ff771bf0c97 in current term 1: Already voted for candidate dca57823003e4729b769ca12367d7530 in this term.
I20240501 20:10:21.770761 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "376ae6034f394a0fb0cab0d8a9241134" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:21.770951 7420 raft_consensus.cc:3037] T 376ae6034f394a0fb0cab0d8a9241134 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:21.771450 9956 leader_election.cc:288] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Requested vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:21.771776 7421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "376ae6034f394a0fb0cab0d8a9241134" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:21.771804 10256 raft_consensus.cc:348] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:21.772063 10256 raft_consensus.cc:374] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:21.772158 10256 raft_consensus.cc:729] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:21.774307 7417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "376ae6034f394a0fb0cab0d8a9241134" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:21.774775 9953 consensus_queue.cc:1035] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:21.775416 10256 consensus_queue.cc:260] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:21.773638 7420 raft_consensus.cc:2454] T 376ae6034f394a0fb0cab0d8a9241134 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:21.776016 10256 ts_tablet_manager.cc:1398] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.009s user 0.001s sys 0.002s
I20240501 20:10:21.775821 9953 raft_consensus.cc:2932] T 264d3fcd3fae4acd8d76306c0f32ed72 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:21.776160 4496 raft_consensus.cc:2932] T 264d3fcd3fae4acd8d76306c0f32ed72 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:21.776813 7223 leader_election.cc:302] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, bceeae1c79964acabdae8ff771bf0c97; no voters: dca57823003e4729b769ca12367d7530
I20240501 20:10:21.779327 9956 raft_consensus.cc:2781] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:21.779510 9956 raft_consensus.cc:686] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Becoming Leader. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Running, Role: LEADER
I20240501 20:10:21.779814 9956 consensus_queue.cc:237] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:21.786433 4091 catalog_manager.cc:5484] T 264d3fcd3fae4acd8d76306c0f32ed72 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:21.788098 10188 raft_consensus.cc:1050] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a: attempting to promote NON_VOTER dca57823003e4729b769ca12367d7530 to VOTER
I20240501 20:10:21.788977 10188 consensus_queue.cc:237] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:21.791978 10075 leader_election.cc:288] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:21.792495 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "376ae6034f394a0fb0cab0d8a9241134" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:21.792483 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "376ae6034f394a0fb0cab0d8a9241134" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:21.792738 4233 raft_consensus.cc:2379] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bceeae1c79964acabdae8ff771bf0c97 in current term 1: Already voted for candidate dca57823003e4729b769ca12367d7530 in this term.
I20240501 20:10:21.793208 8785 leader_election.cc:288] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:21.794811 4168 leader_election.cc:302] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dca57823003e4729b769ca12367d7530; no voters: b9957d4086bc4a76b18ed9265a5739d5, bceeae1c79964acabdae8ff771bf0c97
I20240501 20:10:21.794872 7134 tablet_service.cc:1514] Processing DeleteTablet for tablet 264d3fcd3fae4acd8d76306c0f32ed72 with delete_type TABLET_DATA_TOMBSTONED (TS db0742ee3860468590eefef6ae6074fd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52036
I20240501 20:10:21.797513 7285 consensus_queue.cc:237] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } }
I20240501 20:10:21.798568 7420 raft_consensus.cc:2932] T 264d3fcd3fae4acd8d76306c0f32ed72 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:21.801378 7155 raft_consensus.cc:1261] T c1e7ac38dc9c44d29e2d8127904407eb P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:21.805457 10324 consensus_queue.cc:1035] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:10:21.801082 6958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.72:39817, user_credentials={real_user=slave}} blocked reactor thread for 58913.5us
I20240501 20:10:21.807968 10314 ts_tablet_manager.cc:1398] T 3bdd6de7ac584032aa309269bd0770cf P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.100s user 0.002s sys 0.000s
I20240501 20:10:21.796989 8785 raft_consensus.cc:2726] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 20:10:21.809999 10324 raft_consensus.cc:2932] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:21.815270 4077 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c1e7ac38dc9c44d29e2d8127904407eb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:21.817330 7152 raft_consensus.cc:2932] T c1e7ac38dc9c44d29e2d8127904407eb P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:21.823348 10313 tablet.cc:1620] T 0e85e0affad146d8a7c6b6751012dc70 P 1d34e18a1d394d3f91c9f50d1c7bea4e: MemRowSet was empty: no flush needed.
I20240501 20:10:21.824085 4365 raft_consensus.cc:1261] T c1e7ac38dc9c44d29e2d8127904407eb P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:21.734928 10317 ts_tablet_manager.cc:1398] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.021s user 0.003s sys 0.000s
I20240501 20:10:21.827333 4092 catalog_manager.cc:5484] T c1e7ac38dc9c44d29e2d8127904407eb P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
W20240501 20:10:21.827646 10328 log.cc:924] Time spent T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5: Append to log took a long time: real 0.149s user 0.000s sys 0.000s
I20240501 20:10:21.827620 10324 consensus_queue.cc:1035] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:21.831053 10346 tablet_replica.cc:330] T 264d3fcd3fae4acd8d76306c0f32ed72 P db0742ee3860468590eefef6ae6074fd: stopping tablet replica
I20240501 20:10:21.831406 10346 raft_consensus.cc:2229] T 264d3fcd3fae4acd8d76306c0f32ed72 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:21.831858 10346 raft_consensus.cc:2258] T 264d3fcd3fae4acd8d76306c0f32ed72 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:21.834249 7285 raft_consensus.cc:1261] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:21.835254 10342 consensus_queue.cc:1035] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:21.835920 4496 raft_consensus.cc:1261] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:21.836428 8785 consensus_queue.cc:1035] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:21.837561 4365 raft_consensus.cc:1203] T 61af956bc72b4e3e994a7daf58d1a699 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:21.839277 7022 raft_consensus.cc:1261] T dfaa3794205d482c90aa4f860fa99cf3 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:21.841282 8785 consensus_queue.cc:1035] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:21.841570 4363 raft_consensus.cc:2932] T c1e7ac38dc9c44d29e2d8127904407eb P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:21.842476 10346 ts_tablet_manager.cc:1875] T 264d3fcd3fae4acd8d76306c0f32ed72 P db0742ee3860468590eefef6ae6074fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:21.847889 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 264d3fcd3fae4acd8d76306c0f32ed72 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:21.850911 4234 raft_consensus.cc:1261] T 5a7b2f01b5684df5800987591a390a17 P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:21.851541 10144 consensus_queue.cc:1035] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:21.853209 7155 raft_consensus.cc:1261] T 5a7b2f01b5684df5800987591a390a17 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:21.853380 10255 tablet_bootstrap.cc:492] T 9709a8bb107742b79078f91da9fab8f6 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:21.854483 10144 consensus_queue.cc:1035] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:21.858023 10255 ts_tablet_manager.cc:1367] T 9709a8bb107742b79078f91da9fab8f6 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.119s user 0.011s sys 0.001s
I20240501 20:10:21.856801 10336 raft_consensus.cc:2932] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:21.858904 7287 raft_consensus.cc:2932] T 129322eab07149d2a6fb7eef96100e7a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:21.860484 10245 ts_tablet_manager.cc:902] T e7f35eafb0be4f4eab73133917b22bbf P dca57823003e4729b769ca12367d7530: Initiating tablet copy from peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:21.860430 4093 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:21.860754 4093 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:21.861900 7019 raft_consensus.cc:1261] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:21.863031 10144 consensus_queue.cc:1035] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:21.868450 7019 raft_consensus.cc:1261] T 129322eab07149d2a6fb7eef96100e7a P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:21.869489 10221 consensus_queue.cc:1035] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:21.871965 10255 raft_consensus.cc:348] T 9709a8bb107742b79078f91da9fab8f6 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:21.875946 10255 raft_consensus.cc:374] T 9709a8bb107742b79078f91da9fab8f6 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:21.876467 10255 raft_consensus.cc:729] T 9709a8bb107742b79078f91da9fab8f6 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:21.877063 10255 consensus_queue.cc:260] T 9709a8bb107742b79078f91da9fab8f6 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:21.878715 4093 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 0e85e0affad146d8a7c6b6751012dc70
I20240501 20:10:21.879560 7134 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e85e0affad146d8a7c6b6751012dc70 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:21 UTC) from {username='slave'} at 127.0.0.1:52036
I20240501 20:10:21.880164 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e85e0affad146d8a7c6b6751012dc70 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:21 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:21.880787 7400 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e85e0affad146d8a7c6b6751012dc70 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:21 UTC) from {username='slave'} at 127.0.0.1:40182
I20240501 20:10:21.887094 10144 raft_consensus.cc:2932] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:21.893066 10245 tablet_copy_client.cc:308] T e7f35eafb0be4f4eab73133917b22bbf P dca57823003e4729b769ca12367d7530: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.71:39447
I20240501 20:10:21.894091 10255 ts_tablet_manager.cc:1398] T 9709a8bb107742b79078f91da9fab8f6 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.036s user 0.001s sys 0.001s
I20240501 20:10:21.897147 7152 raft_consensus.cc:2932] T 5a7b2f01b5684df5800987591a390a17 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:21.898787 7155 raft_consensus.cc:1261] T 79af181b5f014a40a4c123b47a720dec P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:21.903242 4496 raft_consensus.cc:1261] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:21.904038 10063 consensus_queue.cc:1035] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:21.905103 10063 consensus_queue.cc:1035] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:21.905323 4627 raft_consensus.cc:2932] T 129322eab07149d2a6fb7eef96100e7a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:21.914000 4363 raft_consensus.cc:1261] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:21.917346 10145 consensus_queue.cc:1035] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:21.917696 4091 catalog_manager.cc:5484] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:21.917608 4233 raft_consensus.cc:2932] T 5a7b2f01b5684df5800987591a390a17 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:21.921392 7401 tablet_service.cc:1514] Processing DeleteTablet for tablet c1e7ac38dc9c44d29e2d8127904407eb with delete_type TABLET_DATA_TOMBSTONED (TS b9957d4086bc4a76b18ed9265a5739d5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40182
I20240501 20:10:21.922751 10252 raft_consensus.cc:482] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:21.922784 4367 consensus_queue.cc:237] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:21.922935 10252 raft_consensus.cc:504] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:21.923554 10252 leader_election.cc:288] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:21.924501 7021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bdd6de7ac584032aa309269bd0770cf" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:21.924752 7021 raft_consensus.cc:2454] T 3bdd6de7ac584032aa309269bd0770cf P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:21.924795 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bdd6de7ac584032aa309269bd0770cf" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:21.925033 4234 raft_consensus.cc:2454] T 3bdd6de7ac584032aa309269bd0770cf P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:21.925494 7090 leader_election.cc:302] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db0742ee3860468590eefef6ae6074fd, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:21.926558 10252 raft_consensus.cc:2781] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:21.926726 10252 raft_consensus.cc:482] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:21.926860 10252 raft_consensus.cc:3037] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:21.931035 10252 raft_consensus.cc:504] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:21.933077 7021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bdd6de7ac584032aa309269bd0770cf" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:21.933319 7021 raft_consensus.cc:3037] T 3bdd6de7ac584032aa309269bd0770cf P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:21.933527 4232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bdd6de7ac584032aa309269bd0770cf" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:10:21.935472 4232 raft_consensus.cc:3037] T 3bdd6de7ac584032aa309269bd0770cf P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:21.937871 7021 raft_consensus.cc:2454] T 3bdd6de7ac584032aa309269bd0770cf P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:21.939968 10358 tablet_replica.cc:330] T 0e85e0affad146d8a7c6b6751012dc70 P b9957d4086bc4a76b18ed9265a5739d5: stopping tablet replica
I20240501 20:10:21.940232 4232 raft_consensus.cc:2454] T 3bdd6de7ac584032aa309269bd0770cf P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:21.940517 10358 raft_consensus.cc:2229] T 0e85e0affad146d8a7c6b6751012dc70 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:21.940830 7090 leader_election.cc:302] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db0742ee3860468590eefef6ae6074fd, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:21.941180 10358 raft_consensus.cc:2258] T 0e85e0affad146d8a7c6b6751012dc70 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:21.941406 10277 raft_consensus.cc:2781] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:21.941592 10277 raft_consensus.cc:686] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Becoming Leader. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Running, Role: LEADER
I20240501 20:10:21.941969 10277 consensus_queue.cc:237] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:21.942322 10358 ts_tablet_manager.cc:1875] T 0e85e0affad146d8a7c6b6751012dc70 P b9957d4086bc4a76b18ed9265a5739d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:21.944917 7019 raft_consensus.cc:2932] T 129322eab07149d2a6fb7eef96100e7a P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:21.947181 10252 leader_election.cc:288] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:21.954325 10338 raft_consensus.cc:2932] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:21.956629 4365 raft_consensus.cc:2932] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:21.956984 7308 tablet_copy_service.cc:140] P bceeae1c79964acabdae8ff771bf0c97: Received BeginTabletCopySession request for tablet e7f35eafb0be4f4eab73133917b22bbf from peer dca57823003e4729b769ca12367d7530 ({username='slave'} at 127.29.219.65:33781)
I20240501 20:10:21.965375 7153 raft_consensus.cc:2932] T 79af181b5f014a40a4c123b47a720dec P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:21.967110 7154 raft_consensus.cc:1261] T 5a7b2f01b5684df5800987591a390a17 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:21.973714 4091 catalog_manager.cc:5484] T 129322eab07149d2a6fb7eef96100e7a P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:21.976146 10346 ts_tablet_manager.cc:1888] T 264d3fcd3fae4acd8d76306c0f32ed72 P db0742ee3860468590eefef6ae6074fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:21.979727 10144 consensus_queue.cc:1035] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:21.981424 4092 catalog_manager.cc:5484] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:21.986254 10346 log.cc:1196] T 264d3fcd3fae4acd8d76306c0f32ed72 P db0742ee3860468590eefef6ae6074fd: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/wals/264d3fcd3fae4acd8d76306c0f32ed72
I20240501 20:10:21.987972 10253 raft_consensus.cc:2932] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:21.989811 4076 catalog_manager.cc:4830] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): tablet 264d3fcd3fae4acd8d76306c0f32ed72 (table rebalancer_extra_table_18 [id=4b4b9eb576884393874a7d89667da324]) successfully deleted
I20240501 20:10:21.987758 10346 tablet_replica.cc:330] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd: stopping tablet replica
I20240501 20:10:21.990299 10346 raft_consensus.cc:2229] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:21.991568 4498 raft_consensus.cc:2932] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:21.992498 10346 raft_consensus.cc:2258] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:21.994885 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5a7b2f01b5684df5800987591a390a17 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:21.999791 7153 raft_consensus.cc:2932] T 5a7b2f01b5684df5800987591a390a17 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:21.999068 4092 catalog_manager.cc:5484] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:22.004071 4089 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:22.010035 4628 consensus_queue.cc:237] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:22.017041 10346 ts_tablet_manager.cc:1875] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:22.017395 7002 tablet_service.cc:1467] Processing CreateTablet for tablet 99e1f9ab5a53461492e1bc548ccf24fc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=99a6f997b9bb42faa6afe18e6e0dc6fc]), partition=
I20240501 20:10:22.017756 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e1f9ab5a53461492e1bc548ccf24fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:22.017827 7308 tablet_copy_service.cc:161] P bceeae1c79964acabdae8ff771bf0c97: Beginning new tablet copy session on tablet e7f35eafb0be4f4eab73133917b22bbf from peer dca57823003e4729b769ca12367d7530 at {username='slave'} at 127.29.219.65:33781: session id = dca57823003e4729b769ca12367d7530-e7f35eafb0be4f4eab73133917b22bbf
I20240501 20:10:22.025271 10314 tablet_bootstrap.cc:492] T 99e1f9ab5a53461492e1bc548ccf24fc P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:22.026881 10314 tablet_bootstrap.cc:654] T 99e1f9ab5a53461492e1bc548ccf24fc P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:22.029388 7400 tablet_service.cc:1467] Processing CreateTablet for tablet 99e1f9ab5a53461492e1bc548ccf24fc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=99a6f997b9bb42faa6afe18e6e0dc6fc]), partition=
I20240501 20:10:22.029759 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e1f9ab5a53461492e1bc548ccf24fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:22.033780 7267 tablet_service.cc:1467] Processing CreateTablet for tablet 99e1f9ab5a53461492e1bc548ccf24fc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=99a6f997b9bb42faa6afe18e6e0dc6fc]), partition=
I20240501 20:10:22.034314 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e1f9ab5a53461492e1bc548ccf24fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:22.034525 7002 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a7b2f01b5684df5800987591a390a17 with delete_type TABLET_DATA_TOMBSTONED (TS ef0a4ada548446b3a531cb39a3c4e38d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:22.045344 7308 tablet_copy_source_session.cc:205] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:22.046938 10245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7f35eafb0be4f4eab73133917b22bbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:22.049444 4089 catalog_manager.cc:5484] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd reported cstate change: term changed from 0 to 1, leader changed from <none> to db0742ee3860468590eefef6ae6074fd (127.29.219.70). New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:22.056146 9953 raft_consensus.cc:2932] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:22.057070 10245 tablet_copy_client.cc:783] T e7f35eafb0be4f4eab73133917b22bbf P dca57823003e4729b769ca12367d7530: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:22.057301 10245 tablet_copy_client.cc:647] T e7f35eafb0be4f4eab73133917b22bbf P dca57823003e4729b769ca12367d7530: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:22.060930 10245 tablet_copy_client.cc:523] T e7f35eafb0be4f4eab73133917b22bbf P dca57823003e4729b769ca12367d7530: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:22.070108 4499 raft_consensus.cc:2932] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:22.078830 7021 consensus_queue.cc:237] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:22.079878 4089 catalog_manager.cc:5484] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 7 to 8, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:22.080250 4087 catalog_manager.cc:5484] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71). New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:22.081404 4365 raft_consensus.cc:1261] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:22.082023 10368 consensus_queue.cc:1035] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:22.082451 4496 raft_consensus.cc:1261] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:22.083539 10368 consensus_queue.cc:1035] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:22.085613 10368 raft_consensus.cc:2932] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:22.087394 4367 raft_consensus.cc:2932] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:22.089936 4496 raft_consensus.cc:2932] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:22.094597 10357 tablet_replica.cc:330] T 0e85e0affad146d8a7c6b6751012dc70 P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:22.096477 10317 tablet_bootstrap.cc:492] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:22.095216 10357 raft_consensus.cc:2229] T 0e85e0affad146d8a7c6b6751012dc70 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:22.097524 10277 raft_consensus.cc:482] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:22.097685 10277 raft_consensus.cc:504] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:22.098399 10317 tablet_bootstrap.cc:654] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:22.098316 10357 raft_consensus.cc:2258] T 0e85e0affad146d8a7c6b6751012dc70 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:22.098968 10277 leader_election.cc:288] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:22.099066 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9709a8bb107742b79078f91da9fab8f6" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:22.099568 7419 raft_consensus.cc:2454] T 9709a8bb107742b79078f91da9fab8f6 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:22.102449 7153 raft_consensus.cc:1261] T 41d6e6dce35e4d0da594f147a7f9bfc4 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:22.104610 7091 leader_election.cc:302] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:22.105486 10252 raft_consensus.cc:2781] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:22.105556 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 79af181b5f014a40a4c123b47a720dec with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:22.105629 10252 raft_consensus.cc:482] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:22.105741 10252 raft_consensus.cc:3037] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:22.108413 10252 raft_consensus.cc:504] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:22.113917 4367 raft_consensus.cc:1261] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 20:10:22.114962 7222 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.67:34595, user_credentials={real_user=slave}} blocked reactor thread for 76557.5us
I20240501 20:10:22.115465 10245 tablet_bootstrap.cc:492] T e7f35eafb0be4f4eab73133917b22bbf P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:22.114966 7287 raft_consensus.cc:2932] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:22.098733 10313 tablet.cc:1722] T 0e85e0affad146d8a7c6b6751012dc70 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240501 20:10:22.102115 4563 consensus_peers.cc:485] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Couldn't send request to peer bceeae1c79964acabdae8ff771bf0c97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41d6e6dce35e4d0da594f147a7f9bfc4. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:22.122563 10255 tablet_bootstrap.cc:492] T 99e1f9ab5a53461492e1bc548ccf24fc P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:22.123263 10317 tablet_bootstrap.cc:492] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:22.123507 10317 ts_tablet_manager.cc:1367] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.004s
I20240501 20:10:22.124305 10255 tablet_bootstrap.cc:654] T 99e1f9ab5a53461492e1bc548ccf24fc P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:22.124418 10317 raft_consensus.cc:348] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:22.124680 10317 raft_consensus.cc:374] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:22.124763 10317 raft_consensus.cc:729] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:22.125082 10317 consensus_queue.cc:260] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:22.125985 10317 ts_tablet_manager.cc:1398] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:10:22.103053 10275 consensus_queue.cc:1035] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:22.126905 10274 consensus_queue.cc:1035] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:22.137161 7022 raft_consensus.cc:2932] T dfaa3794205d482c90aa4f860fa99cf3 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:22.142309 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9709a8bb107742b79078f91da9fab8f6" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:22.142532 7420 raft_consensus.cc:3037] T 9709a8bb107742b79078f91da9fab8f6 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:22.146589 7420 raft_consensus.cc:2454] T 9709a8bb107742b79078f91da9fab8f6 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:22.147181 7091 leader_election.cc:302] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:22.147924 10277 raft_consensus.cc:2781] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:22.148733 10277 raft_consensus.cc:686] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Becoming Leader. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Running, Role: LEADER
I20240501 20:10:22.149360 10358 ts_tablet_manager.cc:1888] T 0e85e0affad146d8a7c6b6751012dc70 P b9957d4086bc4a76b18ed9265a5739d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:22.149298 10277 consensus_queue.cc:237] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:22.149565 10358 log.cc:1196] T 0e85e0affad146d8a7c6b6751012dc70 P b9957d4086bc4a76b18ed9265a5739d5: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/wals/0e85e0affad146d8a7c6b6751012dc70
I20240501 20:10:22.150206 10358 ts_tablet_manager.cc:1909] T 0e85e0affad146d8a7c6b6751012dc70 P b9957d4086bc4a76b18ed9265a5739d5: Deleting consensus metadata
I20240501 20:10:22.150487 4628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9709a8bb107742b79078f91da9fab8f6" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:10:22.150723 4628 raft_consensus.cc:3037] T 9709a8bb107742b79078f91da9fab8f6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:22.151561 10358 tablet_replica.cc:330] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5: stopping tablet replica
I20240501 20:10:22.151619 4079 catalog_manager.cc:4830] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): tablet 0e85e0affad146d8a7c6b6751012dc70 (table rebalancer_dynamic_table_new_name [id=756dbeb8af6e453793147c0af8465921]) successfully deleted
I20240501 20:10:22.152139 10358 raft_consensus.cc:2229] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:22.154062 10252 leader_election.cc:288] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Requested vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:22.159940 4234 raft_consensus.cc:1261] T 5a7b2f01b5684df5800987591a390a17 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:22.161389 10144 consensus_queue.cc:1035] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:22.163058 4234 raft_consensus.cc:2932] T 5a7b2f01b5684df5800987591a390a17 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:22.168656 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9709a8bb107742b79078f91da9fab8f6" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:22.173885 10376 raft_consensus.cc:2932] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:22.174307 4367 raft_consensus.cc:2932] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:22.178488 10372 tablet_replica.cc:330] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d: stopping tablet replica
I20240501 20:10:22.178805 10372 raft_consensus.cc:2229] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:22.179098 10372 pending_rounds.cc:62] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d: Trying to abort 1 pending ops.
I20240501 20:10:22.180083 4628 raft_consensus.cc:2454] T 9709a8bb107742b79078f91da9fab8f6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:22.185520 10372 pending_rounds.cc:68] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7022977932445450240 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "5a7b2f01b5684df5800987591a390a17" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } } }
I20240501 20:10:22.186409 10372 raft_consensus.cc:2894] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240501 20:10:22.186652 10372 raft_consensus.cc:2866] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:10:22.186805 10372 raft_consensus.cc:2258] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:22.187321 4091 catalog_manager.cc:5484] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:22.194329 10314 tablet_bootstrap.cc:492] T 99e1f9ab5a53461492e1bc548ccf24fc P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:22.201339 10314 ts_tablet_manager.cc:1367] T 99e1f9ab5a53461492e1bc548ccf24fc P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.176s user 0.002s sys 0.010s
I20240501 20:10:22.202441 10314 raft_consensus.cc:348] T 99e1f9ab5a53461492e1bc548ccf24fc P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:22.185139 10255 tablet_bootstrap.cc:492] T 99e1f9ab5a53461492e1bc548ccf24fc P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:22.203197 10255 ts_tablet_manager.cc:1367] T 99e1f9ab5a53461492e1bc548ccf24fc P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.081s user 0.006s sys 0.006s
I20240501 20:10:22.204155 10255 raft_consensus.cc:348] T 99e1f9ab5a53461492e1bc548ccf24fc P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:22.207237 10255 raft_consensus.cc:374] T 99e1f9ab5a53461492e1bc548ccf24fc P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:22.207301 7134 tablet_service.cc:1514] Processing DeleteTablet for tablet 79af181b5f014a40a4c123b47a720dec with delete_type TABLET_DATA_TOMBSTONED (TS db0742ee3860468590eefef6ae6074fd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52036
I20240501 20:10:22.208681 4089 catalog_manager.cc:5484] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd reported cstate change: term changed from 0 to 1, leader changed from <none> to db0742ee3860468590eefef6ae6074fd (127.29.219.70). New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:22.211704 4085 catalog_manager.cc:5484] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index 6 to 7, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
W20240501 20:10:22.181915 4562 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.70:36321, user_credentials={real_user=slave}} blocked reactor thread for 67162.7us
W20240501 20:10:22.182991 4563 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.71:39447, user_credentials={real_user=slave}} blocked reactor thread for 58940.3us
I20240501 20:10:22.218143 10357 ts_tablet_manager.cc:1875] T 0e85e0affad146d8a7c6b6751012dc70 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:22.230955 10228 raft_consensus.cc:1050] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b: attempting to promote NON_VOTER 10b47782558c41ee91f5e164a372049a to VOTER
I20240501 20:10:22.231778 10228 consensus_queue.cc:237] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } }
I20240501 20:10:22.232978 10357 ts_tablet_manager.cc:1888] T 0e85e0affad146d8a7c6b6751012dc70 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:22.233119 10357 log.cc:1196] T 0e85e0affad146d8a7c6b6751012dc70 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/0e85e0affad146d8a7c6b6751012dc70
I20240501 20:10:22.235229 4233 raft_consensus.cc:1261] T 61af956bc72b4e3e994a7daf58d1a699 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:22.235443 4367 raft_consensus.cc:1261] T 61af956bc72b4e3e994a7daf58d1a699 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:22.235949 9940 consensus_queue.cc:1035] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:22.236521 9940 consensus_queue.cc:1035] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:22.238669 10245 tablet_bootstrap.cc:492] T e7f35eafb0be4f4eab73133917b22bbf P dca57823003e4729b769ca12367d7530: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:22.239109 10245 tablet_bootstrap.cc:492] T e7f35eafb0be4f4eab73133917b22bbf P dca57823003e4729b769ca12367d7530: Bootstrap complete.
I20240501 20:10:22.239380 10245 ts_tablet_manager.cc:1367] T e7f35eafb0be4f4eab73133917b22bbf P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.124s user 0.031s sys 0.018s
I20240501 20:10:22.240209 10245 raft_consensus.cc:348] T e7f35eafb0be4f4eab73133917b22bbf P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:22.240535 10245 raft_consensus.cc:729] T e7f35eafb0be4f4eab73133917b22bbf P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: LEARNER
I20240501 20:10:22.202766 10314 raft_consensus.cc:374] T 99e1f9ab5a53461492e1bc548ccf24fc P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:22.244408 10314 raft_consensus.cc:729] T 99e1f9ab5a53461492e1bc548ccf24fc P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:22.234884 7420 raft_consensus.cc:1261] T 61af956bc72b4e3e994a7daf58d1a699 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:22.246074 10228 consensus_queue.cc:1035] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:22.247256 10357 ts_tablet_manager.cc:1909] T 0e85e0affad146d8a7c6b6751012dc70 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:22.247575 10245 consensus_queue.cc:260] T e7f35eafb0be4f4eab73133917b22bbf P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:22.248075 10075 raft_consensus.cc:482] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:22.248231 10075 raft_consensus.cc:504] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:22.248930 10075 leader_election.cc:288] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:22.249352 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet 0e85e0affad146d8a7c6b6751012dc70 (table rebalancer_dynamic_table_new_name [id=756dbeb8af6e453793147c0af8465921]) successfully deleted
I20240501 20:10:22.236017 4628 raft_consensus.cc:1203] T 9e31f8d1809244efae05ab9254c163ce P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:22.249673 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e1f9ab5a53461492e1bc548ccf24fc" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:22.250550 10245 ts_tablet_manager.cc:1398] T e7f35eafb0be4f4eab73133917b22bbf P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240501 20:10:22.254977 7308 tablet_copy_service.cc:342] P bceeae1c79964acabdae8ff771bf0c97: Request end of tablet copy session dca57823003e4729b769ca12367d7530-e7f35eafb0be4f4eab73133917b22bbf received from {username='slave'} at 127.29.219.65:33781
I20240501 20:10:22.255136 7308 tablet_copy_service.cc:433] P bceeae1c79964acabdae8ff771bf0c97: ending tablet copy session dca57823003e4729b769ca12367d7530-e7f35eafb0be4f4eab73133917b22bbf on tablet e7f35eafb0be4f4eab73133917b22bbf with peer dca57823003e4729b769ca12367d7530
I20240501 20:10:22.266940 10255 raft_consensus.cc:729] T 99e1f9ab5a53461492e1bc548ccf24fc P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:22.267419 10255 consensus_queue.cc:260] T 99e1f9ab5a53461492e1bc548ccf24fc P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:22.267813 7419 raft_consensus.cc:2454] T 99e1f9ab5a53461492e1bc548ccf24fc P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:22.268644 7223 leader_election.cc:302] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:22.269069 10075 raft_consensus.cc:2781] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:22.269201 10075 raft_consensus.cc:482] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:22.269271 10075 raft_consensus.cc:3037] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:22.270932 9940 raft_consensus.cc:2932] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:22.272444 10075 raft_consensus.cc:504] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:22.274178 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e1f9ab5a53461492e1bc548ccf24fc" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:22.274374 7419 raft_consensus.cc:3037] T 99e1f9ab5a53461492e1bc548ccf24fc P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:22.277753 7419 raft_consensus.cc:2454] T 99e1f9ab5a53461492e1bc548ccf24fc P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:22.279001 4233 raft_consensus.cc:2932] T 61af956bc72b4e3e994a7daf58d1a699 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:22.279266 4089 catalog_manager.cc:5484] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:22.279610 7223 leader_election.cc:302] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:22.279925 10373 raft_consensus.cc:2781] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:22.280040 10373 raft_consensus.cc:686] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Becoming Leader. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Running, Role: LEADER
I20240501 20:10:22.280288 10373 consensus_queue.cc:237] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:22.278168 7420 raft_consensus.cc:2932] T 61af956bc72b4e3e994a7daf58d1a699 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:22.290409 4089 catalog_manager.cc:5484] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71). New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:22.293926 10346 ts_tablet_manager.cc:1888] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:22.305375 10346 log.cc:1196] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/wals/0e85e0affad146d8a7c6b6751012dc70
I20240501 20:10:22.305481 10075 leader_election.cc:288] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:22.306576 10346 ts_tablet_manager.cc:1909] T 0e85e0affad146d8a7c6b6751012dc70 P db0742ee3860468590eefef6ae6074fd: Deleting consensus metadata
I20240501 20:10:22.309628 7419 consensus_queue.cc:237] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } }
I20240501 20:10:22.311970 4628 raft_consensus.cc:1261] T 129322eab07149d2a6fb7eef96100e7a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:22.315058 10337 consensus_queue.cc:1035] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:22.319105 10336 raft_consensus.cc:2932] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:22.322572 4628 raft_consensus.cc:2932] T 129322eab07149d2a6fb7eef96100e7a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:22.325917 4079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 129322eab07149d2a6fb7eef96100e7a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:22.330785 4091 catalog_manager.cc:5484] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:22.332599 4076 catalog_manager.cc:4830] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): tablet 0e85e0affad146d8a7c6b6751012dc70 (table rebalancer_dynamic_table_new_name [id=756dbeb8af6e453793147c0af8465921]) successfully deleted
I20240501 20:10:22.336890 10358 pending_rounds.cc:62] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5: Trying to abort 1 pending ops.
I20240501 20:10:22.337359 10358 pending_rounds.cc:68] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7022977931651379200 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "c1e7ac38dc9c44d29e2d8127904407eb" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } } }
I20240501 20:10:22.338025 10358 raft_consensus.cc:2894] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240501 20:10:22.338222 10358 raft_consensus.cc:2866] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:10:22.339625 10075 consensus_queue.cc:1035] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:22.265342 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e1f9ab5a53461492e1bc548ccf24fc" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:22.274394 7021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e1f9ab5a53461492e1bc548ccf24fc" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:22.312510 7020 raft_consensus.cc:1261] T 129322eab07149d2a6fb7eef96100e7a P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:22.352092 10314 consensus_queue.cc:260] T 99e1f9ab5a53461492e1bc548ccf24fc P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:22.352483 7019 raft_consensus.cc:2454] T 99e1f9ab5a53461492e1bc548ccf24fc P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:22.356492 10346 tablet_replica.cc:330] T 79af181b5f014a40a4c123b47a720dec P db0742ee3860468590eefef6ae6074fd: stopping tablet replica
I20240501 20:10:22.357053 10346 raft_consensus.cc:2229] T 79af181b5f014a40a4c123b47a720dec P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:22.361995 10346 raft_consensus.cc:2258] T 79af181b5f014a40a4c123b47a720dec P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:22.363245 10372 ts_tablet_manager.cc:1875] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:22.367489 4234 raft_consensus.cc:1203] T e7f35eafb0be4f4eab73133917b22bbf P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:22.367578 10346 ts_tablet_manager.cc:1875] T 79af181b5f014a40a4c123b47a720dec P db0742ee3860468590eefef6ae6074fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:22.373485 7153 raft_consensus.cc:2932] T 41d6e6dce35e4d0da594f147a7f9bfc4 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:22.375787 4365 raft_consensus.cc:2932] T 61af956bc72b4e3e994a7daf58d1a699 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:22.377656 10277 consensus_queue.cc:1035] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:22.378733 10314 ts_tablet_manager.cc:1398] T 99e1f9ab5a53461492e1bc548ccf24fc P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.177s user 0.000s sys 0.002s
I20240501 20:10:22.351590 10336 consensus_queue.cc:1035] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:22.380698 7020 raft_consensus.cc:2932] T 129322eab07149d2a6fb7eef96100e7a P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:22.389350 10358 raft_consensus.cc:2258] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:22.378294 10255 ts_tablet_manager.cc:1398] T 99e1f9ab5a53461492e1bc548ccf24fc P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.175s user 0.001s sys 0.001s
I20240501 20:10:22.407207 4496 consensus_queue.cc:237] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } }
I20240501 20:10:22.407645 7267 tablet_service.cc:1514] Processing DeleteTablet for tablet 129322eab07149d2a6fb7eef96100e7a with delete_type TABLET_DATA_TOMBSTONED (TS bceeae1c79964acabdae8ff771bf0c97 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54166
I20240501 20:10:22.409673 4367 raft_consensus.cc:1261] T 61af956bc72b4e3e994a7daf58d1a699 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:22.409780 4234 raft_consensus.cc:1261] T 61af956bc72b4e3e994a7daf58d1a699 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:22.410550 10389 consensus_queue.cc:1035] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:22.410969 10390 consensus_queue.cc:1035] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:22.413848 10230 raft_consensus.cc:2932] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:22.415048 4234 raft_consensus.cc:2932] T 61af956bc72b4e3e994a7daf58d1a699 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:22.418175 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 61af956bc72b4e3e994a7daf58d1a699 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:22.421816 10346 ts_tablet_manager.cc:1888] T 79af181b5f014a40a4c123b47a720dec P db0742ee3860468590eefef6ae6074fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:22.421969 10346 log.cc:1196] T 79af181b5f014a40a4c123b47a720dec P db0742ee3860468590eefef6ae6074fd: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/wals/79af181b5f014a40a4c123b47a720dec
I20240501 20:10:22.422796 4076 catalog_manager.cc:4830] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): tablet 79af181b5f014a40a4c123b47a720dec (table rebalancer_extra_table_30 [id=affa3c54d7ce469c819876e9e5aedb60]) successfully deleted
I20240501 20:10:22.422866 10075 consensus_queue.cc:1035] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:22.424139 4367 raft_consensus.cc:2932] T 61af956bc72b4e3e994a7daf58d1a699 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:22.424484 4093 catalog_manager.cc:5484] T 61af956bc72b4e3e994a7daf58d1a699 P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:22.431038 7400 tablet_service.cc:1514] Processing DeleteTablet for tablet 61af956bc72b4e3e994a7daf58d1a699 with delete_type TABLET_DATA_TOMBSTONED (TS b9957d4086bc4a76b18ed9265a5739d5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40182
I20240501 20:10:22.442448 10358 ts_tablet_manager.cc:1875] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:22.441742 10293 tablet_bootstrap.cc:492] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:22.446405 10293 tablet_bootstrap.cc:492] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd: Bootstrap complete.
I20240501 20:10:22.447273 10293 ts_tablet_manager.cc:1367] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.698s user 0.058s sys 0.008s
I20240501 20:10:22.453699 10399 tablet_replica.cc:330] T 129322eab07149d2a6fb7eef96100e7a P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:22.453950 10399 raft_consensus.cc:2229] T 129322eab07149d2a6fb7eef96100e7a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:22.455189 10399 raft_consensus.cc:2258] T 129322eab07149d2a6fb7eef96100e7a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:22.462256 10399 ts_tablet_manager.cc:1875] T 129322eab07149d2a6fb7eef96100e7a P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:22.465556 10372 ts_tablet_manager.cc:1888] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:22.465920 10372 log.cc:1196] T 5a7b2f01b5684df5800987591a390a17 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/wals/5a7b2f01b5684df5800987591a390a17
I20240501 20:10:22.466889 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet 5a7b2f01b5684df5800987591a390a17 (table rebalancer_extra_table_30 [id=affa3c54d7ce469c819876e9e5aedb60]) successfully deleted
I20240501 20:10:22.461199 10293 raft_consensus.cc:348] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:22.469537 10293 raft_consensus.cc:729] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: LEARNER
I20240501 20:10:22.470139 10293 consensus_queue.cc:260] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:22.473315 10293 ts_tablet_manager.cc:1398] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240501 20:10:22.476380 10399 ts_tablet_manager.cc:1888] T 129322eab07149d2a6fb7eef96100e7a P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:22.477110 4388 tablet_copy_service.cc:342] P 10b47782558c41ee91f5e164a372049a: Request end of tablet copy session db0742ee3860468590eefef6ae6074fd-c66514deec5744da832bc030a582c46a received from {username='slave'} at 127.29.219.70:42453
I20240501 20:10:22.477257 4388 tablet_copy_service.cc:433] P 10b47782558c41ee91f5e164a372049a: ending tablet copy session db0742ee3860468590eefef6ae6074fd-c66514deec5744da832bc030a582c46a on tablet c66514deec5744da832bc030a582c46a with peer db0742ee3860468590eefef6ae6074fd
I20240501 20:10:22.477442 10399 log.cc:1196] T 129322eab07149d2a6fb7eef96100e7a P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/129322eab07149d2a6fb7eef96100e7a
I20240501 20:10:22.480079 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet 129322eab07149d2a6fb7eef96100e7a (table rebalancer_extra_table_28 [id=d841e2e5903247b28bb65617a30e22cd]) successfully deleted
I20240501 20:10:22.482781 10252 consensus_queue.cc:1035] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:22.516180 10358 ts_tablet_manager.cc:1888] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:22.517378 10358 log.cc:1196] T c1e7ac38dc9c44d29e2d8127904407eb P b9957d4086bc4a76b18ed9265a5739d5: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/wals/c1e7ac38dc9c44d29e2d8127904407eb
I20240501 20:10:22.521772 4079 catalog_manager.cc:4830] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): tablet c1e7ac38dc9c44d29e2d8127904407eb (table rebalancer_extra_table_34 [id=513576dc865743a18e99f37632b2a4a2]) successfully deleted
I20240501 20:10:22.528162 10358 tablet_replica.cc:330] T 61af956bc72b4e3e994a7daf58d1a699 P b9957d4086bc4a76b18ed9265a5739d5: stopping tablet replica
I20240501 20:10:22.529536 10358 raft_consensus.cc:2229] T 61af956bc72b4e3e994a7daf58d1a699 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:22.529839 10358 raft_consensus.cc:2258] T 61af956bc72b4e3e994a7daf58d1a699 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:22.530869 10358 ts_tablet_manager.cc:1875] T 61af956bc72b4e3e994a7daf58d1a699 P b9957d4086bc4a76b18ed9265a5739d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:22.537987 10358 ts_tablet_manager.cc:1888] T 61af956bc72b4e3e994a7daf58d1a699 P b9957d4086bc4a76b18ed9265a5739d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:22.538197 10358 log.cc:1196] T 61af956bc72b4e3e994a7daf58d1a699 P b9957d4086bc4a76b18ed9265a5739d5: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/wals/61af956bc72b4e3e994a7daf58d1a699
I20240501 20:10:22.539034 4079 catalog_manager.cc:4830] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): tablet 61af956bc72b4e3e994a7daf58d1a699 (table rebalancer_extra_table_14 [id=be6eb76edc624f598b8543519c4d4476]) successfully deleted
I20240501 20:10:22.544572 7019 consensus_queue.cc:237] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
W20240501 20:10:22.547403 6957 consensus_peers.cc:485] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7afb8e3154a64be78b8041b137b3681a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:22.547506 7287 raft_consensus.cc:1261] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:22.548188 10338 consensus_queue.cc:1035] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:22.549553 4631 raft_consensus.cc:1261] T 7afb8e3154a64be78b8041b137b3681a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:22.550050 10368 consensus_queue.cc:1035] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:22.554699 10368 raft_consensus.cc:2932] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:22.555423 7287 raft_consensus.cc:2932] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:22.562963 4088 catalog_manager.cc:5484] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:22.575104 4628 raft_consensus.cc:2932] T 7afb8e3154a64be78b8041b137b3681a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:22.590498 4234 consensus_queue.cc:237] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
W20240501 20:10:22.593386 4170 consensus_peers.cc:485] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 -> Peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): Couldn't send request to peer b9957d4086bc4a76b18ed9265a5739d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e84a2462cb7e4e559fba887790a65a1f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:22.593408 4498 raft_consensus.cc:1261] T e84a2462cb7e4e559fba887790a65a1f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:22.593842 7153 raft_consensus.cc:1261] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:22.594293 10350 consensus_queue.cc:1035] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:22.594578 9953 consensus_queue.cc:1035] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:22.599465 10350 raft_consensus.cc:2932] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:22.603375 4496 raft_consensus.cc:2932] T e84a2462cb7e4e559fba887790a65a1f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:22.606743 4085 catalog_manager.cc:5484] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index -1 to 2, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:22.617030 10277 consensus_queue.cc:1035] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:22.618664 9926 consensus_queue.cc:1035] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:22.618938 7154 raft_consensus.cc:1203] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:10:22.622491 4088 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:22.636898 7019 raft_consensus.cc:3037] T 99e1f9ab5a53461492e1bc548ccf24fc P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:22.641860 7019 raft_consensus.cc:1261] T 99e1f9ab5a53461492e1bc548ccf24fc P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:22.642537 10075 consensus_queue.cc:1035] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:22.647965 7153 raft_consensus.cc:2932] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:22.649888 10337 raft_consensus.cc:1050] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5: attempting to promote NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e to VOTER
I20240501 20:10:22.650758 10337 consensus_queue.cc:237] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } }
I20240501 20:10:22.656229 7417 raft_consensus.cc:1261] T 99e1f9ab5a53461492e1bc548ccf24fc P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:22.656998 10075 consensus_queue.cc:1035] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:22.674127 4088 catalog_manager.cc:5576] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817) does not have the latest schema for tablet 99e1f9ab5a53461492e1bc548ccf24fc (table rebalancer_dynamic_table [id=99a6f997b9bb42faa6afe18e6e0dc6fc]). Expected version 1 got 0
I20240501 20:10:22.676129 4631 raft_consensus.cc:1261] T 9e31f8d1809244efae05ab9254c163ce P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:22.676631 7285 raft_consensus.cc:1261] T 9e31f8d1809244efae05ab9254c163ce P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:22.677268 10046 consensus_queue.cc:1035] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:22.678524 10221 consensus_queue.cc:1035] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:22.684034 4234 raft_consensus.cc:1261] T 9e31f8d1809244efae05ab9254c163ce P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:22.684688 10221 consensus_queue.cc:1035] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:22.702414 7153 consensus_queue.cc:237] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:22.708070 7022 raft_consensus.cc:1261] T 030335599b754c81810fe0d139ef51f1 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:22.709116 7090 consensus_peers.cc:485] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 030335599b754c81810fe0d139ef51f1. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:22.709343 7417 raft_consensus.cc:1261] T 030335599b754c81810fe0d139ef51f1 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:22.709430 9926 consensus_queue.cc:1035] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:22.709910 10277 consensus_queue.cc:1035] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:22.711450 10434 tablet.cc:1722] T 99e1f9ab5a53461492e1bc548ccf24fc P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:22.711985 10434 tablet.cc:1620] T 99e1f9ab5a53461492e1bc548ccf24fc P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:22.715483 10277 raft_consensus.cc:2932] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:22.717728 4088 catalog_manager.cc:5576] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817) does not have the latest schema for tablet 99e1f9ab5a53461492e1bc548ccf24fc (table rebalancer_dynamic_table [id=99a6f997b9bb42faa6afe18e6e0dc6fc]). Expected version 1 got 0
I20240501 20:10:22.720921 7420 raft_consensus.cc:2932] T 030335599b754c81810fe0d139ef51f1 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:22.727614 10337 raft_consensus.cc:2932] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:22.728091 4088 catalog_manager.cc:5484] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:22.735090 4089 catalog_manager.cc:5576] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237) does not have the latest schema for tablet 99e1f9ab5a53461492e1bc548ccf24fc (table rebalancer_dynamic_table [id=99a6f997b9bb42faa6afe18e6e0dc6fc]). Expected version 1 got 0
I20240501 20:10:22.735838 7022 raft_consensus.cc:2932] T 030335599b754c81810fe0d139ef51f1 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:22.740058 4234 raft_consensus.cc:2932] T 9e31f8d1809244efae05ab9254c163ce P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:22.746696 4631 raft_consensus.cc:2932] T 9e31f8d1809244efae05ab9254c163ce P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:22.749359 4093 catalog_manager.cc:5484] T 9e31f8d1809244efae05ab9254c163ce P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:22.762642 7417 consensus_queue.cc:237] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } }
I20240501 20:10:22.764493 10432 tablet.cc:1722] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:22.767874 10432 tablet.cc:1620] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:22.770452 4234 raft_consensus.cc:1261] T 9e31f8d1809244efae05ab9254c163ce P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:22.771332 10221 consensus_queue.cc:1035] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:22.773967 10221 raft_consensus.cc:2932] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:22.777757 10433 ts_tablet_manager.cc:902] T 41d6e6dce35e4d0da594f147a7f9bfc4 P bceeae1c79964acabdae8ff771bf0c97: Initiating tablet copy from peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:22.777842 10397 mvcc.cc:205] Tried to move back new op lower bound from 7022977935914860544 to 7022977934462271488. Current Snapshot: MvccSnapshot[applied={T|T < 7022977935914860544}]
I20240501 20:10:22.780675 4079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9e31f8d1809244efae05ab9254c163ce with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:22.782347 4232 raft_consensus.cc:2932] T 9e31f8d1809244efae05ab9254c163ce P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:22.789539 4088 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=99a6f997b9bb42faa6afe18e6e0dc6fc] alter complete (version 1)
I20240501 20:10:22.793893 4089 catalog_manager.cc:5484] T 9e31f8d1809244efae05ab9254c163ce P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:22.794258 4093 catalog_manager.cc:5576] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237) does not have the latest schema for tablet 99e1f9ab5a53461492e1bc548ccf24fc (table rebalancer_dynamic_table [id=99a6f997b9bb42faa6afe18e6e0dc6fc]). Expected version 1 got 0
I20240501 20:10:22.796329 10447 tablet.cc:1722] T 99e1f9ab5a53461492e1bc548ccf24fc P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:22.797019 10447 tablet.cc:1620] T 99e1f9ab5a53461492e1bc548ccf24fc P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:22.798524 7267 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e31f8d1809244efae05ab9254c163ce with delete_type TABLET_DATA_TOMBSTONED (TS bceeae1c79964acabdae8ff771bf0c97 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54166
I20240501 20:10:22.798887 10399 tablet_replica.cc:330] T 9e31f8d1809244efae05ab9254c163ce P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:22.799118 10399 raft_consensus.cc:2229] T 9e31f8d1809244efae05ab9254c163ce P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:22.799419 10399 pending_rounds.cc:62] T 9e31f8d1809244efae05ab9254c163ce P bceeae1c79964acabdae8ff771bf0c97: Trying to abort 1 pending ops.
I20240501 20:10:22.799484 10399 pending_rounds.cc:68] T 9e31f8d1809244efae05ab9254c163ce P bceeae1c79964acabdae8ff771bf0c97: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7022977935974764544 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "9e31f8d1809244efae05ab9254c163ce" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } } }
I20240501 20:10:22.799927 10399 raft_consensus.cc:2894] T 9e31f8d1809244efae05ab9254c163ce P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240501 20:10:22.800040 10399 raft_consensus.cc:2866] T 9e31f8d1809244efae05ab9254c163ce P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:10:22.800171 10399 raft_consensus.cc:2258] T 9e31f8d1809244efae05ab9254c163ce P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:22.801782 10399 ts_tablet_manager.cc:1875] T 9e31f8d1809244efae05ab9254c163ce P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:22.804102 4631 raft_consensus.cc:1261] T 9e31f8d1809244efae05ab9254c163ce P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:22.804726 10221 consensus_queue.cc:1035] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:22.805719 4093 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:22.806398 4628 raft_consensus.cc:2932] T 9e31f8d1809244efae05ab9254c163ce P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:22.819983 10399 ts_tablet_manager.cc:1888] T 9e31f8d1809244efae05ab9254c163ce P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:22.822558 10399 log.cc:1196] T 9e31f8d1809244efae05ab9254c163ce P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/9e31f8d1809244efae05ab9254c163ce
I20240501 20:10:22.822980 7134 tablet_service.cc:1467] Processing CreateTablet for tablet 99bd0e9f4cd7496785ad71480d7d3d61 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=19fe11bc7ef54ec8be09ca4882586266]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:22.823098 4477 tablet_service.cc:1467] Processing CreateTablet for tablet 99bd0e9f4cd7496785ad71480d7d3d61 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=19fe11bc7ef54ec8be09ca4882586266]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:22.823441 7134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99bd0e9f4cd7496785ad71480d7d3d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:22.823448 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99bd0e9f4cd7496785ad71480d7d3d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:22.825244 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet 9e31f8d1809244efae05ab9254c163ce (table rebalancer_extra_table_20 [id=880fdf37926743948f9ffe03bdaeeaa5]) successfully deleted
I20240501 20:10:22.828521 7002 tablet_service.cc:1467] Processing CreateTablet for tablet 99bd0e9f4cd7496785ad71480d7d3d61 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=19fe11bc7ef54ec8be09ca4882586266]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:22.828974 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99bd0e9f4cd7496785ad71480d7d3d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:22.836457 7001 tablet_service.cc:1467] Processing CreateTablet for tablet cd21a0faa3bb487cb80f0951b12b3d67 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=19fe11bc7ef54ec8be09ca4882586266]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:22.836905 7001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd21a0faa3bb487cb80f0951b12b3d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:22.837908 4346 tablet_service.cc:1467] Processing CreateTablet for tablet cd21a0faa3bb487cb80f0951b12b3d67 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=19fe11bc7ef54ec8be09ca4882586266]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:22.838038 7133 tablet_service.cc:1467] Processing CreateTablet for tablet cd21a0faa3bb487cb80f0951b12b3d67 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=19fe11bc7ef54ec8be09ca4882586266]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:22.838297 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd21a0faa3bb487cb80f0951b12b3d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:22.838549 7133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd21a0faa3bb487cb80f0951b12b3d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:22.844235 10451 tablet_bootstrap.cc:492] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:22.846375 10451 tablet_bootstrap.cc:654] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:22.846634 10314 tablet_bootstrap.cc:492] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:22.848232 10314 tablet_bootstrap.cc:654] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:22.849112 4477 tablet_service.cc:1467] Processing CreateTablet for tablet 56b2082919b548bfb56b939373caf770 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=19fe11bc7ef54ec8be09ca4882586266]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:22.852967 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56b2082919b548bfb56b939373caf770. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:22.855418 7135 tablet_service.cc:1467] Processing CreateTablet for tablet 56b2082919b548bfb56b939373caf770 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=19fe11bc7ef54ec8be09ca4882586266]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:22.856343 7001 tablet_service.cc:1467] Processing CreateTablet for tablet 56b2082919b548bfb56b939373caf770 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=19fe11bc7ef54ec8be09ca4882586266]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:22.857398 7001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56b2082919b548bfb56b939373caf770. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:22.859344 7135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56b2082919b548bfb56b939373caf770. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:22.867444 10454 tablet_bootstrap.cc:492] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:22.871768 10456 tablet_bootstrap.cc:492] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:22.872862 10314 tablet_bootstrap.cc:492] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:22.873040 10314 ts_tablet_manager.cc:1367] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.009s
I20240501 20:10:22.874166 10314 raft_consensus.cc:348] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:22.874958 10314 raft_consensus.cc:374] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:22.875442 10451 tablet_bootstrap.cc:492] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:10:22.875622 10451 ts_tablet_manager.cc:1367] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.000s
I20240501 20:10:22.875154 10314 raft_consensus.cc:729] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:22.876166 10314 consensus_queue.cc:260] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:22.876443 10451 raft_consensus.cc:348] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:22.876688 10451 raft_consensus.cc:374] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:22.876817 10451 raft_consensus.cc:729] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:10:22.877094 10451 consensus_queue.cc:260] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:22.880091 10451 ts_tablet_manager.cc:1398] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:10:22.880399 10451 tablet_bootstrap.cc:492] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:22.881819 10451 tablet_bootstrap.cc:654] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:22.883929 10456 tablet_bootstrap.cc:654] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:22.883562 10314 ts_tablet_manager.cc:1398] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.010s user 0.001s sys 0.002s
I20240501 20:10:22.893215 10314 tablet_bootstrap.cc:492] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:22.896355 4089 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:22.900744 10451 tablet_bootstrap.cc:492] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:10:22.901008 10451 ts_tablet_manager.cc:1367] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.006s
I20240501 20:10:22.901904 10451 raft_consensus.cc:348] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:22.904218 10451 raft_consensus.cc:374] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:22.904918 10451 raft_consensus.cc:729] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:10:22.905273 10451 consensus_queue.cc:260] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:22.907213 10451 ts_tablet_manager.cc:1398] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 20:10:22.909598 10454 tablet_bootstrap.cc:654] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:22.914690 10314 tablet_bootstrap.cc:654] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:22.917279 10432 tablet.cc:1722] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:22.917954 10432 tablet.cc:1620] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:22.920867 4631 consensus_queue.cc:237] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:22.922886 10447 tablet.cc:1722] T 99e1f9ab5a53461492e1bc548ccf24fc P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:22.923435 10447 tablet.cc:1620] T 99e1f9ab5a53461492e1bc548ccf24fc P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:22.924237 10463 ts_tablet_manager.cc:902] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b: Initiating tablet copy from peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:22.926606 10434 tablet.cc:1722] T 99e1f9ab5a53461492e1bc548ccf24fc P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:22.927132 10434 tablet.cc:1620] T 99e1f9ab5a53461492e1bc548ccf24fc P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:22.934036 4088 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=99a6f997b9bb42faa6afe18e6e0dc6fc] alter complete (version 2)
I20240501 20:10:22.934929 10314 tablet_bootstrap.cc:492] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:22.935225 10314 ts_tablet_manager.cc:1367] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.001s
I20240501 20:10:22.937141 10433 tablet_copy_client.cc:308] T 41d6e6dce35e4d0da594f147a7f9bfc4 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.68:45205
I20240501 20:10:22.938094 10454 tablet_bootstrap.cc:492] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:10:22.938114 4651 tablet_copy_service.cc:140] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Received BeginTabletCopySession request for tablet 41d6e6dce35e4d0da594f147a7f9bfc4 from peer bceeae1c79964acabdae8ff771bf0c97 ({username='slave'} at 127.29.219.71:48361)
I20240501 20:10:22.938238 10454 ts_tablet_manager.cc:1367] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.000s
I20240501 20:10:22.938264 4651 tablet_copy_service.cc:161] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Beginning new tablet copy session on tablet 41d6e6dce35e4d0da594f147a7f9bfc4 from peer bceeae1c79964acabdae8ff771bf0c97 at {username='slave'} at 127.29.219.71:48361: session id = bceeae1c79964acabdae8ff771bf0c97-41d6e6dce35e4d0da594f147a7f9bfc4
I20240501 20:10:22.938995 10454 raft_consensus.cc:348] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:22.939209 10454 raft_consensus.cc:374] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:22.939258 10454 raft_consensus.cc:729] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:10:22.939450 10454 consensus_queue.cc:260] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:22.939949 10454 ts_tablet_manager.cc:1398] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:22.941825 4651 tablet_copy_source_session.cc:205] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:22.943362 10433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41d6e6dce35e4d0da594f147a7f9bfc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:22.944515 10443 raft_consensus.cc:1050] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97: attempting to promote NON_VOTER dca57823003e4729b769ca12367d7530 to VOTER
I20240501 20:10:22.945632 10443 consensus_queue.cc:237] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
W20240501 20:10:22.950455 4565 consensus_peers.cc:485] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): Couldn't send request to peer b9957d4086bc4a76b18ed9265a5739d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7725fa9f98e9439797a532ed6f2f7dba. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:22.952131 7153 raft_consensus.cc:1261] T 7725fa9f98e9439797a532ed6f2f7dba P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:22.952839 10382 consensus_queue.cc:1035] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:22.953423 10314 raft_consensus.cc:348] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:22.965827 10456 tablet_bootstrap.cc:492] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:22.966127 10456 ts_tablet_manager.cc:1367] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.094s user 0.013s sys 0.014s
I20240501 20:10:22.968935 10456 raft_consensus.cc:348] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:22.953650 10314 raft_consensus.cc:374] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:22.971017 10314 raft_consensus.cc:729] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:22.971405 10456 raft_consensus.cc:374] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:22.971741 10456 raft_consensus.cc:729] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:22.972167 10456 consensus_queue.cc:260] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:22.973580 10314 consensus_queue.cc:260] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:22.975361 4232 raft_consensus.cc:1261] T e7f35eafb0be4f4eab73133917b22bbf P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:22.977006 10314 ts_tablet_manager.cc:1398] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.042s user 0.002s sys 0.000s
I20240501 20:10:22.977917 10456 ts_tablet_manager.cc:1398] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240501 20:10:22.978286 10314 tablet_bootstrap.cc:492] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:22.978322 10075 consensus_queue.cc:1035] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:22.982769 10456 tablet_bootstrap.cc:492] T 56b2082919b548bfb56b939373caf770 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:22.982321 10382 raft_consensus.cc:2932] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:22.984074 10314 tablet_bootstrap.cc:654] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:22.986675 10456 tablet_bootstrap.cc:654] T 56b2082919b548bfb56b939373caf770 P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:22.995333 7153 raft_consensus.cc:2932] T 7725fa9f98e9439797a532ed6f2f7dba P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:22.997393 10433 tablet_copy_client.cc:783] T 41d6e6dce35e4d0da594f147a7f9bfc4 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:22.997684 10433 tablet_copy_client.cc:647] T 41d6e6dce35e4d0da594f147a7f9bfc4 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:23.003026 4628 raft_consensus.cc:1261] T e7f35eafb0be4f4eab73133917b22bbf P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:23.003654 10443 consensus_queue.cc:1035] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:23.004423 7022 raft_consensus.cc:1261] T e7f35eafb0be4f4eab73133917b22bbf P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:23.005635 10373 consensus_queue.cc:1035] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:23.006243 4234 raft_consensus.cc:1261] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:23.006919 10377 consensus_queue.cc:1035] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:23.009280 4088 catalog_manager.cc:5484] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 6 to 7, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:23.022877 10433 tablet_copy_client.cc:523] T 41d6e6dce35e4d0da594f147a7f9bfc4 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:23.024626 10478 ts_tablet_manager.cc:902] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5: Initiating tablet copy from peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:23.029528 4234 raft_consensus.cc:2932] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:23.036185 4093 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:23.037400 10314 tablet_bootstrap.cc:492] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:23.038724 10373 raft_consensus.cc:2932] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:23.039992 10314 ts_tablet_manager.cc:1367] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.002s
I20240501 20:10:23.049131 10432 tablet.cc:1722] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:23.049592 10338 raft_consensus.cc:482] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:23.049669 4233 raft_consensus.cc:2932] T e7f35eafb0be4f4eab73133917b22bbf P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:23.050335 10314 raft_consensus.cc:348] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:23.050554 10314 raft_consensus.cc:374] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:23.050639 10314 raft_consensus.cc:729] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:23.050915 10314 consensus_queue.cc:260] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:23.049791 10338 raft_consensus.cc:504] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:23.052208 10338 leader_election.cc:288] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:23.053818 10434 tablet.cc:1722] T 99e1f9ab5a53461492e1bc548ccf24fc P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:23.054409 4496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bd0e9f4cd7496785ad71480d7d3d61" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:23.054673 4496 raft_consensus.cc:2454] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:23.055455 6957 leader_election.cc:302] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:23.057500 10314 ts_tablet_manager.cc:1398] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240501 20:10:23.060214 10063 raft_consensus.cc:2781] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:23.060334 10063 raft_consensus.cc:482] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:23.060420 10063 raft_consensus.cc:3037] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:23.061702 4088 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table_new_name [id=99a6f997b9bb42faa6afe18e6e0dc6fc] alter complete (version 3)
I20240501 20:10:23.064738 10253 raft_consensus.cc:1050] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a: attempting to promote NON_VOTER db0742ee3860468590eefef6ae6074fd to VOTER
I20240501 20:10:23.065713 10253 consensus_queue.cc:237] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:23.069517 10063 raft_consensus.cc:504] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:23.070137 10063 leader_election.cc:288] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Requested vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:23.070818 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bd0e9f4cd7496785ad71480d7d3d61" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:10:23.071004 4498 raft_consensus.cc:3037] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:23.071076 7154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bd0e9f4cd7496785ad71480d7d3d61" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:23.071300 7154 raft_consensus.cc:3037] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:23.073535 4498 raft_consensus.cc:2454] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:23.074160 6957 leader_election.cc:302] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:23.074765 10145 raft_consensus.cc:2781] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:23.074931 10145 raft_consensus.cc:686] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Becoming Leader. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Running, Role: LEADER
I20240501 20:10:23.075160 10145 consensus_queue.cc:237] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:23.075453 7154 raft_consensus.cc:2454] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:23.089038 4093 catalog_manager.cc:5484] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69). New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:23.089429 10145 raft_consensus.cc:482] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:23.089594 10145 raft_consensus.cc:504] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:23.090284 10145 leader_election.cc:288] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:23.091112 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b2082919b548bfb56b939373caf770" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:23.091324 4498 raft_consensus.cc:2454] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:23.091362 10478 tablet_copy_client.cc:308] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.65:42941
I20240501 20:10:23.091751 6957 leader_election.cc:302] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:23.092643 4254 tablet_copy_service.cc:140] P dca57823003e4729b769ca12367d7530: Received BeginTabletCopySession request for tablet e84a2462cb7e4e559fba887790a65a1f from peer b9957d4086bc4a76b18ed9265a5739d5 ({username='slave'} at 127.29.219.72:48035)
I20240501 20:10:23.092810 4254 tablet_copy_service.cc:161] P dca57823003e4729b769ca12367d7530: Beginning new tablet copy session on tablet e84a2462cb7e4e559fba887790a65a1f from peer b9957d4086bc4a76b18ed9265a5739d5 at {username='slave'} at 127.29.219.72:48035: session id = b9957d4086bc4a76b18ed9265a5739d5-e84a2462cb7e4e559fba887790a65a1f
I20240501 20:10:23.093027 4628 raft_consensus.cc:2932] T e7f35eafb0be4f4eab73133917b22bbf P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:23.096513 4254 tablet_copy_source_session.cc:205] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:23.098018 10063 raft_consensus.cc:2781] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:23.098119 10063 raft_consensus.cc:482] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:23.098215 10063 raft_consensus.cc:3037] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:23.099551 10478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e84a2462cb7e4e559fba887790a65a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.103101 10063 raft_consensus.cc:504] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:23.104986 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b2082919b548bfb56b939373caf770" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:10:23.105247 4498 raft_consensus.cc:3037] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:23.108057 4498 raft_consensus.cc:2454] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:23.109548 6957 leader_election.cc:302] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:23.109968 10478 tablet_copy_client.cc:783] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:23.110057 10338 raft_consensus.cc:2781] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:23.110244 10338 raft_consensus.cc:686] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Becoming Leader. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Running, Role: LEADER
I20240501 20:10:23.110723 10447 tablet.cc:1722] T 99e1f9ab5a53461492e1bc548ccf24fc P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:23.110574 10338 consensus_queue.cc:237] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:23.117516 7019 raft_consensus.cc:2932] T e7f35eafb0be4f4eab73133917b22bbf P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:23.117642 4093 catalog_manager.cc:5484] T e7f35eafb0be4f4eab73133917b22bbf P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:23.122510 7154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b2082919b548bfb56b939373caf770" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
W20240501 20:10:23.123307 6955 leader_election.cc:341] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:23.123689 10433 tablet_bootstrap.cc:492] T 41d6e6dce35e4d0da594f147a7f9bfc4 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:23.124469 10368 raft_consensus.cc:482] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:23.124676 10368 raft_consensus.cc:504] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:23.125288 10368 leader_election.cc:288] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:23.126183 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd21a0faa3bb487cb80f0951b12b3d67" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:10:23.126396 4365 raft_consensus.cc:2454] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:23.126907 6957 leader_election.cc:302] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:23.127223 10368 raft_consensus.cc:2781] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:23.127323 10368 raft_consensus.cc:482] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:23.127441 10368 raft_consensus.cc:3037] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:23.128156 10063 leader_election.cc:288] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:23.128463 7154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd21a0faa3bb487cb80f0951b12b3d67" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:23.128531 7152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b2082919b548bfb56b939373caf770" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
W20240501 20:10:23.129215 6955 leader_election.cc:341] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:10:23.131163 6955 leader_election.cc:341] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:23.130163 10368 raft_consensus.cc:504] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:23.131865 10478 tablet_copy_client.cc:647] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:23.132167 10368 leader_election.cc:288] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Requested vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:23.132576 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd21a0faa3bb487cb80f0951b12b3d67" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:10:23.132794 4365 raft_consensus.cc:3037] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:23.132710 7154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd21a0faa3bb487cb80f0951b12b3d67" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
W20240501 20:10:23.133241 6955 leader_election.cc:341] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:23.135344 4365 raft_consensus.cc:2454] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:23.135363 7287 consensus_queue.cc:237] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:23.136308 6957 leader_election.cc:302] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, ef0a4ada548446b3a531cb39a3c4e38d; no voters: db0742ee3860468590eefef6ae6074fd
I20240501 20:10:23.136827 10063 raft_consensus.cc:2781] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:23.137382 10063 raft_consensus.cc:686] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Becoming Leader. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Running, Role: LEADER
I20240501 20:10:23.137791 10063 consensus_queue.cc:237] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:23.140437 4232 raft_consensus.cc:1261] T e7f35eafb0be4f4eab73133917b22bbf P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:23.141567 4628 raft_consensus.cc:1261] T e7f35eafb0be4f4eab73133917b22bbf P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:23.142679 10478 tablet_copy_client.cc:523] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:23.142858 10075 consensus_queue.cc:1035] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:23.146850 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bd0e9f4cd7496785ad71480d7d3d61" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:23.147105 7153 raft_consensus.cc:2362] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1. Re-sending same reply.
I20240501 20:10:23.153398 10075 consensus_queue.cc:1035] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20240501 20:10:23.158702 9956 raft_consensus.cc:2932] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:23.165818 4077 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e7f35eafb0be4f4eab73133917b22bbf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:23.170257 4627 raft_consensus.cc:2932] T e7f35eafb0be4f4eab73133917b22bbf P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:23.174191 10456 tablet_bootstrap.cc:492] T 56b2082919b548bfb56b939373caf770 P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:23.175303 10478 tablet_bootstrap.cc:492] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:23.175983 10456 ts_tablet_manager.cc:1367] T 56b2082919b548bfb56b939373caf770 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.193s user 0.011s sys 0.017s
I20240501 20:10:23.179967 4088 catalog_manager.cc:5484] T e7f35eafb0be4f4eab73133917b22bbf P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:23.181582 10456 raft_consensus.cc:348] T 56b2082919b548bfb56b939373caf770 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:23.188742 4496 raft_consensus.cc:1261] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:23.189772 10253 consensus_queue.cc:1035] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:23.191956 7001 tablet_service.cc:1514] Processing DeleteTablet for tablet e7f35eafb0be4f4eab73133917b22bbf with delete_type TABLET_DATA_TOMBSTONED (TS ef0a4ada548446b3a531cb39a3c4e38d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:23.192737 10497 tablet_replica.cc:330] T e7f35eafb0be4f4eab73133917b22bbf P ef0a4ada548446b3a531cb39a3c4e38d: stopping tablet replica
I20240501 20:10:23.192962 10497 raft_consensus.cc:2229] T e7f35eafb0be4f4eab73133917b22bbf P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:23.193279 10497 raft_consensus.cc:2258] T e7f35eafb0be4f4eab73133917b22bbf P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:23.194730 10497 ts_tablet_manager.cc:1875] T e7f35eafb0be4f4eab73133917b22bbf P ef0a4ada548446b3a531cb39a3c4e38d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:23.195725 10495 ts_tablet_manager.cc:902] T 030335599b754c81810fe0d139ef51f1 P 10b47782558c41ee91f5e164a372049a: Initiating tablet copy from peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:23.197389 10456 raft_consensus.cc:374] T 56b2082919b548bfb56b939373caf770 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:23.198688 10456 raft_consensus.cc:729] T 56b2082919b548bfb56b939373caf770 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:23.200321 10456 consensus_queue.cc:260] T 56b2082919b548bfb56b939373caf770 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:23.205468 10456 ts_tablet_manager.cc:1398] T 56b2082919b548bfb56b939373caf770 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.029s user 0.001s sys 0.001s
I20240501 20:10:23.207348 4091 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:23.207717 4091 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:23.214320 10456 tablet_bootstrap.cc:492] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:23.215799 4091 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 99e1f9ab5a53461492e1bc548ccf24fc
I20240501 20:10:23.217728 7267 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e1f9ab5a53461492e1bc548ccf24fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:23 UTC) from {username='slave'} at 127.0.0.1:54166
I20240501 20:10:23.218041 10399 tablet_replica.cc:330] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:23.218245 10399 raft_consensus.cc:2229] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:23.218593 10399 raft_consensus.cc:2258] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:23.219172 7400 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e1f9ab5a53461492e1bc548ccf24fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:23 UTC) from {username='slave'} at 127.0.0.1:40182
I20240501 20:10:23.219810 10399 ts_tablet_manager.cc:1875] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:23.220997 7287 raft_consensus.cc:1261] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:23.222318 10188 consensus_queue.cc:1035] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:23.227056 10399 ts_tablet_manager.cc:1888] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:23.227254 10399 log.cc:1196] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/99e1f9ab5a53461492e1bc548ccf24fc
I20240501 20:10:23.227857 10399 ts_tablet_manager.cc:1909] T 99e1f9ab5a53461492e1bc548ccf24fc P bceeae1c79964acabdae8ff771bf0c97: Deleting consensus metadata
I20240501 20:10:23.229108 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet 99e1f9ab5a53461492e1bc548ccf24fc (table rebalancer_dynamic_table_new_name [id=99a6f997b9bb42faa6afe18e6e0dc6fc]) successfully deleted
I20240501 20:10:23.202590 10497 ts_tablet_manager.cc:1888] T e7f35eafb0be4f4eab73133917b22bbf P ef0a4ada548446b3a531cb39a3c4e38d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:23.230209 10497 log.cc:1196] T e7f35eafb0be4f4eab73133917b22bbf P ef0a4ada548446b3a531cb39a3c4e38d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/wals/e7f35eafb0be4f4eab73133917b22bbf
I20240501 20:10:23.217402 7001 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e1f9ab5a53461492e1bc548ccf24fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:23 UTC) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:23.234325 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet e7f35eafb0be4f4eab73133917b22bbf (table rebalancer_extra_table_24 [id=b446a76fc1184ee88b0b983584d3f5f4]) successfully deleted
I20240501 20:10:23.234701 10456 tablet_bootstrap.cc:654] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:23.235363 10497 tablet_replica.cc:330] T 99e1f9ab5a53461492e1bc548ccf24fc P ef0a4ada548446b3a531cb39a3c4e38d: stopping tablet replica
I20240501 20:10:23.235849 10497 raft_consensus.cc:2229] T 99e1f9ab5a53461492e1bc548ccf24fc P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:23.235949 4232 raft_consensus.cc:2932] T e7f35eafb0be4f4eab73133917b22bbf P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:23.237376 10497 raft_consensus.cc:2258] T 99e1f9ab5a53461492e1bc548ccf24fc P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:23.237656 10500 tablet_replica.cc:330] T 99e1f9ab5a53461492e1bc548ccf24fc P b9957d4086bc4a76b18ed9265a5739d5: stopping tablet replica
I20240501 20:10:23.237963 10500 raft_consensus.cc:2229] T 99e1f9ab5a53461492e1bc548ccf24fc P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:23.239533 10497 ts_tablet_manager.cc:1875] T 99e1f9ab5a53461492e1bc548ccf24fc P ef0a4ada548446b3a531cb39a3c4e38d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:23.240113 10500 raft_consensus.cc:2258] T 99e1f9ab5a53461492e1bc548ccf24fc P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:23.244112 10500 ts_tablet_manager.cc:1875] T 99e1f9ab5a53461492e1bc548ccf24fc P b9957d4086bc4a76b18ed9265a5739d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:23.247409 7019 consensus_queue.cc:237] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:23.250010 7153 raft_consensus.cc:1261] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:23.251458 10253 consensus_queue.cc:1035] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:23.254415 10253 raft_consensus.cc:2932] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:23.257948 10478 tablet_bootstrap.cc:492] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:23.259662 10456 tablet_bootstrap.cc:492] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:23.260105 7283 raft_consensus.cc:2932] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:23.262446 4496 raft_consensus.cc:2932] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
W20240501 20:10:23.264081 6957 consensus_peers.cc:485] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f63c61d85524d87a4b911c321f486c4. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:23.263789 10456 ts_tablet_manager.cc:1367] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.000s
I20240501 20:10:23.264597 7287 raft_consensus.cc:1261] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:23.265082 10338 consensus_queue.cc:1035] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:23.268021 4093 catalog_manager.cc:5484] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index 7 to 8, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:23.272063 10497 ts_tablet_manager.cc:1888] T 99e1f9ab5a53461492e1bc548ccf24fc P ef0a4ada548446b3a531cb39a3c4e38d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:23.273495 10497 log.cc:1196] T 99e1f9ab5a53461492e1bc548ccf24fc P ef0a4ada548446b3a531cb39a3c4e38d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/wals/99e1f9ab5a53461492e1bc548ccf24fc
I20240501 20:10:23.273530 4631 raft_consensus.cc:1261] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:23.274152 10497 ts_tablet_manager.cc:1909] T 99e1f9ab5a53461492e1bc548ccf24fc P ef0a4ada548446b3a531cb39a3c4e38d: Deleting consensus metadata
I20240501 20:10:23.274646 10338 consensus_queue.cc:1035] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:23.275368 10478 tablet_bootstrap.cc:492] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap complete.
I20240501 20:10:23.275671 10478 ts_tablet_manager.cc:1367] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.100s user 0.008s sys 0.000s
I20240501 20:10:23.276466 10478 raft_consensus.cc:348] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:23.276741 10478 raft_consensus.cc:729] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: LEARNER
I20240501 20:10:23.276978 10478 consensus_queue.cc:260] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:23.270493 10456 raft_consensus.cc:348] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:23.279683 10456 raft_consensus.cc:374] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:23.280176 10456 raft_consensus.cc:729] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:23.280220 10338 raft_consensus.cc:2932] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:23.281208 10456 consensus_queue.cc:260] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } }
I20240501 20:10:23.284281 4628 raft_consensus.cc:2932] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:23.285214 10456 ts_tablet_manager.cc:1398] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240501 20:10:23.295233 10478 ts_tablet_manager.cc:1398] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240501 20:10:23.296909 4254 tablet_copy_service.cc:342] P dca57823003e4729b769ca12367d7530: Request end of tablet copy session b9957d4086bc4a76b18ed9265a5739d5-e84a2462cb7e4e559fba887790a65a1f received from {username='slave'} at 127.29.219.72:48035
I20240501 20:10:23.297037 4254 tablet_copy_service.cc:433] P dca57823003e4729b769ca12367d7530: ending tablet copy session b9957d4086bc4a76b18ed9265a5739d5-e84a2462cb7e4e559fba887790a65a1f on tablet e84a2462cb7e4e559fba887790a65a1f with peer b9957d4086bc4a76b18ed9265a5739d5
I20240501 20:10:23.297327 10500 ts_tablet_manager.cc:1888] T 99e1f9ab5a53461492e1bc548ccf24fc P b9957d4086bc4a76b18ed9265a5739d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:23.297454 10500 log.cc:1196] T 99e1f9ab5a53461492e1bc548ccf24fc P b9957d4086bc4a76b18ed9265a5739d5: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/wals/99e1f9ab5a53461492e1bc548ccf24fc
I20240501 20:10:23.306217 7154 raft_consensus.cc:2932] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:23.311026 10463 tablet_copy_client.cc:308] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.69:46237
I20240501 20:10:23.311793 7042 tablet_copy_service.cc:140] P ef0a4ada548446b3a531cb39a3c4e38d: Received BeginTabletCopySession request for tablet 7afb8e3154a64be78b8041b137b3681a from peer 3e974c03d3c546f9ab6c91e762a5426b ({username='slave'} at 127.29.219.67:45609)
I20240501 20:10:23.311955 7042 tablet_copy_service.cc:161] P ef0a4ada548446b3a531cb39a3c4e38d: Beginning new tablet copy session on tablet 7afb8e3154a64be78b8041b137b3681a from peer 3e974c03d3c546f9ab6c91e762a5426b at {username='slave'} at 127.29.219.67:45609: session id = 3e974c03d3c546f9ab6c91e762a5426b-7afb8e3154a64be78b8041b137b3681a
I20240501 20:10:23.311080 10500 ts_tablet_manager.cc:1909] T 99e1f9ab5a53461492e1bc548ccf24fc P b9957d4086bc4a76b18ed9265a5739d5: Deleting consensus metadata
I20240501 20:10:23.316200 7042 tablet_copy_source_session.cc:205] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:23.318004 10463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7afb8e3154a64be78b8041b137b3681a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.277022 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet 99e1f9ab5a53461492e1bc548ccf24fc (table rebalancer_dynamic_table_new_name [id=99a6f997b9bb42faa6afe18e6e0dc6fc]) successfully deleted
W20240501 20:10:23.322176 4078 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.29.219.69:46237, user_credentials={real_user=slave}} blocked reactor thread for 45160.5us
I20240501 20:10:23.250274 4091 catalog_manager.cc:5484] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69). New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:23.293975 4085 catalog_manager.cc:5484] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:23.329466 10463 tablet_copy_client.cc:783] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:23.317338 4079 catalog_manager.cc:4830] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): tablet 99e1f9ab5a53461492e1bc548ccf24fc (table rebalancer_dynamic_table_new_name [id=99a6f997b9bb42faa6afe18e6e0dc6fc]) successfully deleted
I20240501 20:10:23.329718 10463 tablet_copy_client.cc:647] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:23.320087 4084 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:23.336829 10463 tablet_copy_client.cc:523] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:23.341533 7001 tablet_service.cc:1514] Processing DeleteTablet for tablet e7f35eafb0be4f4eab73133917b22bbf with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 4)) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:23.341925 7002 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e1f9ab5a53461492e1bc548ccf24fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:23 UTC) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:23.342211 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet e7f35eafb0be4f4eab73133917b22bbf (table rebalancer_extra_table_24 [id=b446a76fc1184ee88b0b983584d3f5f4]) successfully deleted
W20240501 20:10:23.344377 4078 catalog_manager.cc:4797] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): delete failed for tablet 99e1f9ab5a53461492e1bc548ccf24fc because the tablet was not found. No further retry: Not found: Tablet not found: 99e1f9ab5a53461492e1bc548ccf24fc
I20240501 20:10:23.351094 10463 tablet_bootstrap.cc:492] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:23.356892 10495 tablet_copy_client.cc:308] T 030335599b754c81810fe0d139ef51f1 P 10b47782558c41ee91f5e164a372049a: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.70:36321
I20240501 20:10:23.357666 7175 tablet_copy_service.cc:140] P db0742ee3860468590eefef6ae6074fd: Received BeginTabletCopySession request for tablet 030335599b754c81810fe0d139ef51f1 from peer 10b47782558c41ee91f5e164a372049a ({username='slave'} at 127.29.219.66:51075)
I20240501 20:10:23.357904 7175 tablet_copy_service.cc:161] P db0742ee3860468590eefef6ae6074fd: Beginning new tablet copy session on tablet 030335599b754c81810fe0d139ef51f1 from peer 10b47782558c41ee91f5e164a372049a at {username='slave'} at 127.29.219.66:51075: session id = 10b47782558c41ee91f5e164a372049a-030335599b754c81810fe0d139ef51f1
I20240501 20:10:23.361217 7175 tablet_copy_source_session.cc:205] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:23.362210 10478 ts_tablet_manager.cc:902] T 7725fa9f98e9439797a532ed6f2f7dba P b9957d4086bc4a76b18ed9265a5739d5: Initiating tablet copy from peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:23.363927 4087 catalog_manager.cc:5484] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69). New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:23.374380 7287 raft_consensus.cc:2932] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:23.382800 7267 tablet_service.cc:1467] Processing CreateTablet for tablet ecca8fc3a57047c396cd2551df962e5d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=198a355f97e3458f8a2c4889a3091902]), partition=
I20240501 20:10:23.383010 4610 tablet_service.cc:1467] Processing CreateTablet for tablet ecca8fc3a57047c396cd2551df962e5d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=198a355f97e3458f8a2c4889a3091902]), partition=
I20240501 20:10:23.383210 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecca8fc3a57047c396cd2551df962e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.383430 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecca8fc3a57047c396cd2551df962e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.387962 10433 tablet_bootstrap.cc:492] T 41d6e6dce35e4d0da594f147a7f9bfc4 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:23.388445 10433 tablet_bootstrap.cc:492] T 41d6e6dce35e4d0da594f147a7f9bfc4 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap complete.
I20240501 20:10:23.388738 10433 ts_tablet_manager.cc:1367] T 41d6e6dce35e4d0da594f147a7f9bfc4 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.265s user 0.057s sys 0.018s
I20240501 20:10:23.392119 7400 tablet_service.cc:1467] Processing CreateTablet for tablet ecca8fc3a57047c396cd2551df962e5d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=198a355f97e3458f8a2c4889a3091902]), partition=
I20240501 20:10:23.394471 10463 tablet_bootstrap.cc:492] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:23.394708 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecca8fc3a57047c396cd2551df962e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.394852 10463 tablet_bootstrap.cc:492] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap complete.
I20240501 20:10:23.395090 10463 ts_tablet_manager.cc:1367] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.044s user 0.013s sys 0.003s
I20240501 20:10:23.395854 10463 raft_consensus.cc:348] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:23.396356 10463 raft_consensus.cc:729] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: LEARNER
I20240501 20:10:23.396539 10463 consensus_queue.cc:260] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:23.397277 10463 ts_tablet_manager.cc:1398] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:10:23.399013 7042 tablet_copy_service.cc:342] P ef0a4ada548446b3a531cb39a3c4e38d: Request end of tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-7afb8e3154a64be78b8041b137b3681a received from {username='slave'} at 127.29.219.67:45609
I20240501 20:10:23.399179 7042 tablet_copy_service.cc:433] P ef0a4ada548446b3a531cb39a3c4e38d: ending tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-7afb8e3154a64be78b8041b137b3681a on tablet 7afb8e3154a64be78b8041b137b3681a with peer 3e974c03d3c546f9ab6c91e762a5426b
I20240501 20:10:23.389606 10433 raft_consensus.cc:348] T 41d6e6dce35e4d0da594f147a7f9bfc4 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:23.400331 10433 raft_consensus.cc:729] T 41d6e6dce35e4d0da594f147a7f9bfc4 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: LEARNER
I20240501 20:10:23.400549 10433 consensus_queue.cc:260] T 41d6e6dce35e4d0da594f147a7f9bfc4 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:23.401528 10433 ts_tablet_manager.cc:1398] T 41d6e6dce35e4d0da594f147a7f9bfc4 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.013s user 0.003s sys 0.001s
I20240501 20:10:23.402233 4651 tablet_copy_service.cc:342] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Request end of tablet copy session bceeae1c79964acabdae8ff771bf0c97-41d6e6dce35e4d0da594f147a7f9bfc4 received from {username='slave'} at 127.29.219.71:48361
I20240501 20:10:23.402360 4651 tablet_copy_service.cc:433] P 1d34e18a1d394d3f91c9f50d1c7bea4e: ending tablet copy session bceeae1c79964acabdae8ff771bf0c97-41d6e6dce35e4d0da594f147a7f9bfc4 on tablet 41d6e6dce35e4d0da594f147a7f9bfc4 with peer bceeae1c79964acabdae8ff771bf0c97
I20240501 20:10:23.404062 10518 tablet_bootstrap.cc:492] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:23.404750 4365 consensus_queue.cc:237] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:23.406823 7287 raft_consensus.cc:1261] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:23.406985 7154 raft_consensus.cc:1261] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:23.407518 10188 consensus_queue.cc:1035] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:23.409662 10188 consensus_queue.cc:1035] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:23.411151 10517 tablet_bootstrap.cc:492] T ecca8fc3a57047c396cd2551df962e5d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:23.412570 10253 raft_consensus.cc:2932] T c66514deec5744da832bc030a582c46a P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:23.415277 10517 tablet_bootstrap.cc:654] T ecca8fc3a57047c396cd2551df962e5d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:23.416138 7420 raft_consensus.cc:1203] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:23.419260 7154 raft_consensus.cc:2932] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:23.428706 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c66514deec5744da832bc030a582c46a with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:23.432258 7287 raft_consensus.cc:2932] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:23.439373 10518 tablet_bootstrap.cc:654] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:23.439395 4088 catalog_manager.cc:5484] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 8 to 9, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:23.450516 4477 tablet_service.cc:1514] Processing DeleteTablet for tablet c66514deec5744da832bc030a582c46a with delete_type TABLET_DATA_TOMBSTONED (TS 3e974c03d3c546f9ab6c91e762a5426b not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:23.455992 10522 tablet_bootstrap.cc:492] T ecca8fc3a57047c396cd2551df962e5d P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:23.461668 10522 tablet_bootstrap.cc:654] T ecca8fc3a57047c396cd2551df962e5d P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:23.462471 10518 tablet_bootstrap.cc:492] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:23.462755 10518 ts_tablet_manager.cc:1367] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.059s user 0.005s sys 0.000s
I20240501 20:10:23.463714 10518 raft_consensus.cc:348] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:23.464128 10518 raft_consensus.cc:374] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:23.464300 10518 raft_consensus.cc:729] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:23.464609 10518 consensus_queue.cc:260] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:23.465561 10518 ts_tablet_manager.cc:1398] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:23.479754 10495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 030335599b754c81810fe0d139ef51f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.480600 10522 tablet_bootstrap.cc:492] T ecca8fc3a57047c396cd2551df962e5d P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:23.484757 10522 ts_tablet_manager.cc:1367] T ecca8fc3a57047c396cd2551df962e5d P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.004s
I20240501 20:10:23.485738 10522 raft_consensus.cc:348] T ecca8fc3a57047c396cd2551df962e5d P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:23.490414 10522 raft_consensus.cc:374] T ecca8fc3a57047c396cd2551df962e5d P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:23.490629 10522 raft_consensus.cc:729] T ecca8fc3a57047c396cd2551df962e5d P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:23.491243 10522 consensus_queue.cc:260] T ecca8fc3a57047c396cd2551df962e5d P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:23.492612 10522 ts_tablet_manager.cc:1398] T ecca8fc3a57047c396cd2551df962e5d P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 20:10:23.497501 10521 tablet_replica.cc:330] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:23.497987 10521 raft_consensus.cc:2229] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:23.498384 10521 raft_consensus.cc:2258] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:23.502074 10495 tablet_copy_client.cc:783] T 030335599b754c81810fe0d139ef51f1 P 10b47782558c41ee91f5e164a372049a: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:23.502326 10495 tablet_copy_client.cc:647] T 030335599b754c81810fe0d139ef51f1 P 10b47782558c41ee91f5e164a372049a: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:23.504355 10521 ts_tablet_manager.cc:1875] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:23.504400 10495 tablet_copy_client.cc:523] T 030335599b754c81810fe0d139ef51f1 P 10b47782558c41ee91f5e164a372049a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:23.505144 10517 tablet_bootstrap.cc:492] T ecca8fc3a57047c396cd2551df962e5d P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:23.507484 10517 ts_tablet_manager.cc:1367] T ecca8fc3a57047c396cd2551df962e5d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.096s user 0.018s sys 0.000s
I20240501 20:10:23.508587 10517 raft_consensus.cc:348] T ecca8fc3a57047c396cd2551df962e5d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:23.508962 10517 raft_consensus.cc:374] T ecca8fc3a57047c396cd2551df962e5d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:23.509114 10517 raft_consensus.cc:729] T ecca8fc3a57047c396cd2551df962e5d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:23.510272 10517 consensus_queue.cc:260] T ecca8fc3a57047c396cd2551df962e5d P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:23.513773 10521 ts_tablet_manager.cc:1888] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:10:23.513931 10521 log.cc:1196] T c66514deec5744da832bc030a582c46a P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/c66514deec5744da832bc030a582c46a
I20240501 20:10:23.516991 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet c66514deec5744da832bc030a582c46a (table rebalance_test_table_11 [id=3b02a4c6c0f54681a87f5ca2381de01d]) successfully deleted
I20240501 20:10:23.521246 10495 tablet_bootstrap.cc:492] T 030335599b754c81810fe0d139ef51f1 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:23.529096 10517 ts_tablet_manager.cc:1398] T ecca8fc3a57047c396cd2551df962e5d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240501 20:10:23.530836 10338 consensus_queue.cc:1035] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:23.546228 7154 consensus_queue.cc:237] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:23.547947 10478 tablet_copy_client.cc:308] T 7725fa9f98e9439797a532ed6f2f7dba P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.68:45205
I20240501 20:10:23.548755 4651 tablet_copy_service.cc:140] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Received BeginTabletCopySession request for tablet 7725fa9f98e9439797a532ed6f2f7dba from peer b9957d4086bc4a76b18ed9265a5739d5 ({username='slave'} at 127.29.219.72:52769)
I20240501 20:10:23.548969 4651 tablet_copy_service.cc:161] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Beginning new tablet copy session on tablet 7725fa9f98e9439797a532ed6f2f7dba from peer b9957d4086bc4a76b18ed9265a5739d5 at {username='slave'} at 127.29.219.72:52769: session id = b9957d4086bc4a76b18ed9265a5739d5-7725fa9f98e9439797a532ed6f2f7dba
I20240501 20:10:23.562026 7283 raft_consensus.cc:1261] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:23.562455 7091 consensus_peers.cc:485] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd -> Peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Couldn't send request to peer 1d34e18a1d394d3f91c9f50d1c7bea4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c147c98b674e499992f0b9f556ac6815. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:23.562810 10252 consensus_queue.cc:1035] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:23.564258 7019 raft_consensus.cc:1261] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:23.564962 10277 consensus_queue.cc:1035] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:23.576848 7154 raft_consensus.cc:3037] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:23.578294 4651 tablet_copy_source_session.cc:205] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:23.581269 10063 consensus_queue.cc:1035] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:23.584318 10063 consensus_queue.cc:1035] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:23.584923 10495 tablet_bootstrap.cc:492] T 030335599b754c81810fe0d139ef51f1 P 10b47782558c41ee91f5e164a372049a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:23.585300 10495 tablet_bootstrap.cc:492] T 030335599b754c81810fe0d139ef51f1 P 10b47782558c41ee91f5e164a372049a: Bootstrap complete.
I20240501 20:10:23.586731 10495 ts_tablet_manager.cc:1367] T 030335599b754c81810fe0d139ef51f1 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.066s user 0.011s sys 0.017s
I20240501 20:10:23.588982 10495 raft_consensus.cc:348] T 030335599b754c81810fe0d139ef51f1 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:23.589195 10495 raft_consensus.cc:729] T 030335599b754c81810fe0d139ef51f1 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: LEARNER
I20240501 20:10:23.589360 10495 consensus_queue.cc:260] T 030335599b754c81810fe0d139ef51f1 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:23.589866 10495 ts_tablet_manager.cc:1398] T 030335599b754c81810fe0d139ef51f1 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:10:23.593240 7175 tablet_copy_service.cc:342] P db0742ee3860468590eefef6ae6074fd: Request end of tablet copy session 10b47782558c41ee91f5e164a372049a-030335599b754c81810fe0d139ef51f1 received from {username='slave'} at 127.29.219.66:51075
I20240501 20:10:23.593396 7175 tablet_copy_service.cc:433] P db0742ee3860468590eefef6ae6074fd: ending tablet copy session 10b47782558c41ee91f5e164a372049a-030335599b754c81810fe0d139ef51f1 on tablet 030335599b754c81810fe0d139ef51f1 with peer 10b47782558c41ee91f5e164a372049a
I20240501 20:10:23.602144 10368 consensus_queue.cc:1035] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:23.613293 10478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7725fa9f98e9439797a532ed6f2f7dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.620275 10277 raft_consensus.cc:2932] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:23.625442 7020 raft_consensus.cc:2932] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:23.637421 4087 catalog_manager.cc:5484] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:23.656872 10463 ts_tablet_manager.cc:902] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b: Initiating tablet copy from peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:23.662961 7153 raft_consensus.cc:3037] T 56b2082919b548bfb56b939373caf770 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:23.666900 10145 consensus_queue.cc:1035] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:23.668146 7283 raft_consensus.cc:2932] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:23.672020 10478 tablet_copy_client.cc:783] T 7725fa9f98e9439797a532ed6f2f7dba P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:23.672420 10478 tablet_copy_client.cc:647] T 7725fa9f98e9439797a532ed6f2f7dba P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:23.674019 10478 tablet_copy_client.cc:523] T 7725fa9f98e9439797a532ed6f2f7dba P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:23.678022 10478 tablet_bootstrap.cc:492] T 7725fa9f98e9439797a532ed6f2f7dba P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:23.683465 10368 consensus_queue.cc:1035] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:23.704296 10046 raft_consensus.cc:482] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:23.704524 10046 raft_consensus.cc:504] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:23.705232 10046 leader_election.cc:288] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:23.706194 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecca8fc3a57047c396cd2551df962e5d" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:23.706277 4628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecca8fc3a57047c396cd2551df962e5d" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:23.706468 4628 raft_consensus.cc:2454] T ecca8fc3a57047c396cd2551df962e5d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:10:23.706470 7287 raft_consensus.cc:2454] T ecca8fc3a57047c396cd2551df962e5d P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:10:23.706902 7356 leader_election.cc:302] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, b9957d4086bc4a76b18ed9265a5739d5; no voters:
I20240501 20:10:23.707337 10046 raft_consensus.cc:2781] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:23.707420 10046 raft_consensus.cc:482] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:23.707502 10046 raft_consensus.cc:3037] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:23.710233 10046 raft_consensus.cc:504] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:23.711171 10046 leader_election.cc:288] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Requested vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:23.711395 4628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecca8fc3a57047c396cd2551df962e5d" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:10:23.711607 4628 raft_consensus.cc:3037] T ecca8fc3a57047c396cd2551df962e5d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:23.712905 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecca8fc3a57047c396cd2551df962e5d" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:23.713141 7287 raft_consensus.cc:3037] T ecca8fc3a57047c396cd2551df962e5d P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:23.714454 4628 raft_consensus.cc:2454] T ecca8fc3a57047c396cd2551df962e5d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:23.716965 7356 leader_election.cc:302] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, b9957d4086bc4a76b18ed9265a5739d5; no voters:
I20240501 20:10:23.717331 10046 raft_consensus.cc:2781] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:23.717486 10046 raft_consensus.cc:686] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Becoming Leader. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Running, Role: LEADER
I20240501 20:10:23.717753 10046 consensus_queue.cc:237] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } }
I20240501 20:10:23.720414 7287 raft_consensus.cc:2454] T ecca8fc3a57047c396cd2551df962e5d P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:23.734565 4232 consensus_queue.cc:237] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
W20240501 20:10:23.737267 4169 consensus_peers.cc:485] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6cb1ca3dbf614f94b39e8ddd28ab04fc. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:23.737704 7287 raft_consensus.cc:1261] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:23.738313 7155 raft_consensus.cc:1261] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:23.738785 9953 consensus_queue.cc:1035] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:23.739460 9953 consensus_queue.cc:1035] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:23.742401 4087 catalog_manager.cc:5484] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72). New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:23.770453 9953 raft_consensus.cc:2932] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:23.771272 7287 raft_consensus.cc:2932] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:23.773613 10463 tablet_copy_client.cc:308] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.69:46237
I20240501 20:10:23.775693 7042 tablet_copy_service.cc:140] P ef0a4ada548446b3a531cb39a3c4e38d: Received BeginTabletCopySession request for tablet 3f63c61d85524d87a4b911c321f486c4 from peer 3e974c03d3c546f9ab6c91e762a5426b ({username='slave'} at 127.29.219.67:45609)
I20240501 20:10:23.775915 7042 tablet_copy_service.cc:161] P ef0a4ada548446b3a531cb39a3c4e38d: Beginning new tablet copy session on tablet 3f63c61d85524d87a4b911c321f486c4 from peer 3e974c03d3c546f9ab6c91e762a5426b at {username='slave'} at 127.29.219.67:45609: session id = 3e974c03d3c546f9ab6c91e762a5426b-3f63c61d85524d87a4b911c321f486c4
I20240501 20:10:23.779057 4089 catalog_manager.cc:5484] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:23.788105 7042 tablet_copy_source_session.cc:205] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:23.789573 10463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f63c61d85524d87a4b911c321f486c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.797412 4365 raft_consensus.cc:1203] T 030335599b754c81810fe0d139ef51f1 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:23.798055 4498 raft_consensus.cc:1203] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:23.809597 10463 tablet_copy_client.cc:783] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:23.809854 10463 tablet_copy_client.cc:647] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:23.814256 10463 tablet_copy_client.cc:523] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:23.818499 7155 raft_consensus.cc:2932] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:23.827509 10463 tablet_bootstrap.cc:492] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:23.834192 7287 raft_consensus.cc:1203] T 41d6e6dce35e4d0da594f147a7f9bfc4 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:10:23.856688 10463 tablet_bootstrap.cc:492] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:23.857046 10463 tablet_bootstrap.cc:492] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap complete.
I20240501 20:10:23.857256 10463 ts_tablet_manager.cc:1367] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.008s
I20240501 20:10:23.857820 10463 raft_consensus.cc:348] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:23.858000 10463 raft_consensus.cc:729] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: LEARNER
I20240501 20:10:23.858167 10463 consensus_queue.cc:260] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:23.858805 10463 ts_tablet_manager.cc:1398] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 20:10:23.867856 7042 tablet_copy_service.cc:342] P ef0a4ada548446b3a531cb39a3c4e38d: Request end of tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-3f63c61d85524d87a4b911c321f486c4 received from {username='slave'} at 127.29.219.67:45609
I20240501 20:10:23.868005 7042 tablet_copy_service.cc:433] P ef0a4ada548446b3a531cb39a3c4e38d: ending tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-3f63c61d85524d87a4b911c321f486c4 on tablet 3f63c61d85524d87a4b911c321f486c4 with peer 3e974c03d3c546f9ab6c91e762a5426b
I20240501 20:10:23.904222 7019 consensus_queue.cc:237] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:23.909533 4087 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:23.911713 4631 raft_consensus.cc:1261] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:23.912253 7420 raft_consensus.cc:1261] T 743d7fd2ae014352a5681c2a8ee4447d P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:23.913133 10145 consensus_queue.cc:1035] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:23.913697 10145 consensus_queue.cc:1035] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:23.916376 6957 consensus_peers.cc:485] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 743d7fd2ae014352a5681c2a8ee4447d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:23.922288 10368 raft_consensus.cc:2932] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:23.923262 7417 raft_consensus.cc:2932] T 743d7fd2ae014352a5681c2a8ee4447d P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:23.924803 10478 tablet_bootstrap.cc:492] T 7725fa9f98e9439797a532ed6f2f7dba P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:23.925232 10478 tablet_bootstrap.cc:492] T 7725fa9f98e9439797a532ed6f2f7dba P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap complete.
I20240501 20:10:23.925506 10478 ts_tablet_manager.cc:1367] T 7725fa9f98e9439797a532ed6f2f7dba P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.248s user 0.049s sys 0.014s
I20240501 20:10:23.926378 10478 raft_consensus.cc:348] T 7725fa9f98e9439797a532ed6f2f7dba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:23.926750 10478 raft_consensus.cc:729] T 7725fa9f98e9439797a532ed6f2f7dba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: LEARNER
I20240501 20:10:23.926945 10478 consensus_queue.cc:260] T 7725fa9f98e9439797a532ed6f2f7dba P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:23.927695 10478 ts_tablet_manager.cc:1398] T 7725fa9f98e9439797a532ed6f2f7dba P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:10:23.929613 4651 tablet_copy_service.cc:342] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Request end of tablet copy session b9957d4086bc4a76b18ed9265a5739d5-7725fa9f98e9439797a532ed6f2f7dba received from {username='slave'} at 127.29.219.72:52769
I20240501 20:10:23.929769 4651 tablet_copy_service.cc:433] P 1d34e18a1d394d3f91c9f50d1c7bea4e: ending tablet copy session b9957d4086bc4a76b18ed9265a5739d5-7725fa9f98e9439797a532ed6f2f7dba on tablet 7725fa9f98e9439797a532ed6f2f7dba with peer b9957d4086bc4a76b18ed9265a5739d5
I20240501 20:10:23.930055 4087 catalog_manager.cc:5484] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:23.942368 4631 raft_consensus.cc:1261] T ecca8fc3a57047c396cd2551df962e5d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:23.942669 7287 raft_consensus.cc:1261] T ecca8fc3a57047c396cd2551df962e5d P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:23.942902 10337 consensus_queue.cc:1035] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:23.943341 10221 consensus_queue.cc:1035] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:23.944087 4628 raft_consensus.cc:2932] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:23.949564 10576 tablet.cc:1722] T ecca8fc3a57047c396cd2551df962e5d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:23.950222 10576 tablet.cc:1620] T ecca8fc3a57047c396cd2551df962e5d P 1d34e18a1d394d3f91c9f50d1c7bea4e: MemRowSet was empty: no flush needed.
I20240501 20:10:23.950999 4087 catalog_manager.cc:5576] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205) does not have the latest schema for tablet ecca8fc3a57047c396cd2551df962e5d (table rebalancer_dynamic_table [id=198a355f97e3458f8a2c4889a3091902]). Expected version 1 got 0
I20240501 20:10:23.954988 4089 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:23.967515 4088 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=198a355f97e3458f8a2c4889a3091902] alter complete (version 1)
I20240501 20:10:23.969975 10575 tablet.cc:1722] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:23.970598 4088 catalog_manager.cc:5576] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817) does not have the latest schema for tablet ecca8fc3a57047c396cd2551df962e5d (table rebalancer_dynamic_table [id=198a355f97e3458f8a2c4889a3091902]). Expected version 1 got 0
I20240501 20:10:23.970803 10575 tablet.cc:1620] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:23.973312 7002 tablet_service.cc:1467] Processing CreateTablet for tablet accaff8513134f96b5502b5c5385d892 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8cce50d935840afb7fe769c351116d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.973942 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet accaff8513134f96b5502b5c5385d892. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.975560 7400 tablet_service.cc:1467] Processing CreateTablet for tablet accaff8513134f96b5502b5c5385d892 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8cce50d935840afb7fe769c351116d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.976003 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet accaff8513134f96b5502b5c5385d892. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.982590 7401 tablet_service.cc:1467] Processing CreateTablet for tablet 786296114e8a47c68df103934ec055d5 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8cce50d935840afb7fe769c351116d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.982745 10578 tablet_bootstrap.cc:492] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:23.983443 4215 tablet_service.cc:1467] Processing CreateTablet for tablet accaff8513134f96b5502b5c5385d892 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8cce50d935840afb7fe769c351116d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.983870 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet accaff8513134f96b5502b5c5385d892. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.984054 4477 tablet_service.cc:1467] Processing CreateTablet for tablet 786296114e8a47c68df103934ec055d5 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8cce50d935840afb7fe769c351116d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.984413 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 786296114e8a47c68df103934ec055d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.987039 7267 tablet_service.cc:1467] Processing CreateTablet for tablet 786296114e8a47c68df103934ec055d5 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8cce50d935840afb7fe769c351116d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.987460 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 786296114e8a47c68df103934ec055d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.987929 7266 tablet_service.cc:1467] Processing CreateTablet for tablet cf05708a20a04c66875867f5453f7fb9 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8cce50d935840afb7fe769c351116d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.988340 7266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf05708a20a04c66875867f5453f7fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.983122 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 786296114e8a47c68df103934ec055d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:23.991480 4346 tablet_service.cc:1467] Processing CreateTablet for tablet cf05708a20a04c66875867f5453f7fb9 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8cce50d935840afb7fe769c351116d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:23.991868 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf05708a20a04c66875867f5453f7fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:24.004015 4087 catalog_manager.cc:5576] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447) does not have the latest schema for tablet ecca8fc3a57047c396cd2551df962e5d (table rebalancer_dynamic_table [id=198a355f97e3458f8a2c4889a3091902]). Expected version 1 got 0
I20240501 20:10:24.003464 10578 tablet_bootstrap.cc:654] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:24.005038 7400 tablet_service.cc:1467] Processing CreateTablet for tablet cf05708a20a04c66875867f5453f7fb9 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=b8cce50d935840afb7fe769c351116d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:24.005929 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf05708a20a04c66875867f5453f7fb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:24.006804 10580 tablet_bootstrap.cc:492] T accaff8513134f96b5502b5c5385d892 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:24.009375 10580 tablet_bootstrap.cc:654] T accaff8513134f96b5502b5c5385d892 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:24.021467 9802 raft_consensus.cc:1050] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530: attempting to promote NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 to VOTER
I20240501 20:10:24.022475 9802 consensus_queue.cc:237] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:24.029886 7155 raft_consensus.cc:1261] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:24.030685 10350 consensus_queue.cc:1035] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:24.032874 4498 raft_consensus.cc:1261] T e84a2462cb7e4e559fba887790a65a1f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:24.033625 10350 consensus_queue.cc:1035] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:24.033913 7417 raft_consensus.cc:1261] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:24.034497 10350 consensus_queue.cc:1035] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:24.065922 10584 tablet_bootstrap.cc:492] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:24.068186 10584 tablet_bootstrap.cc:654] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:24.069898 9953 raft_consensus.cc:2932] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:24.070446 10583 tablet_bootstrap.cc:492] T cf05708a20a04c66875867f5453f7fb9 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:24.072132 10583 tablet_bootstrap.cc:654] T cf05708a20a04c66875867f5453f7fb9 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:24.073683 4087 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:24.084468 10575 tablet.cc:1722] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:24.085065 10575 tablet.cc:1620] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:24.086333 4498 raft_consensus.cc:2932] T e84a2462cb7e4e559fba887790a65a1f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:24.086920 10576 tablet.cc:1722] T ecca8fc3a57047c396cd2551df962e5d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:24.087801 10576 tablet.cc:1620] T ecca8fc3a57047c396cd2551df962e5d P 1d34e18a1d394d3f91c9f50d1c7bea4e: MemRowSet was empty: no flush needed.
I20240501 20:10:24.092413 7155 raft_consensus.cc:2932] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:24.097126 10579 tablet_bootstrap.cc:492] T accaff8513134f96b5502b5c5385d892 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:24.099352 10579 tablet_bootstrap.cc:654] T accaff8513134f96b5502b5c5385d892 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:24.100651 10582 tablet_bootstrap.cc:492] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:24.101892 4088 catalog_manager.cc:5484] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 2 to 3, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:24.102332 10582 tablet_bootstrap.cc:654] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:24.105468 10587 tablet.cc:1722] T ecca8fc3a57047c396cd2551df962e5d P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:24.109021 10587 tablet.cc:1620] T ecca8fc3a57047c396cd2551df962e5d P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:24.109807 4091 catalog_manager.cc:5576] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447) does not have the latest schema for tablet ecca8fc3a57047c396cd2551df962e5d (table rebalancer_dynamic_table [id=198a355f97e3458f8a2c4889a3091902]). Expected version 2 got 0
I20240501 20:10:24.116406 4087 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=198a355f97e3458f8a2c4889a3091902] alter complete (version 2)
I20240501 20:10:24.118778 10583 tablet_bootstrap.cc:492] T cf05708a20a04c66875867f5453f7fb9 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:10:24.118999 10583 ts_tablet_manager.cc:1367] T cf05708a20a04c66875867f5453f7fb9 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
I20240501 20:10:24.119908 10583 raft_consensus.cc:348] T cf05708a20a04c66875867f5453f7fb9 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:24.120235 10583 raft_consensus.cc:374] T cf05708a20a04c66875867f5453f7fb9 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:24.120361 10583 raft_consensus.cc:729] T cf05708a20a04c66875867f5453f7fb9 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:10:24.120638 10583 consensus_queue.cc:260] T cf05708a20a04c66875867f5453f7fb9 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:24.122968 10583 ts_tablet_manager.cc:1398] T cf05708a20a04c66875867f5453f7fb9 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 20:10:24.131660 10584 tablet_bootstrap.cc:492] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:24.131920 10584 ts_tablet_manager.cc:1367] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.066s user 0.006s sys 0.000s
I20240501 20:10:24.132915 10584 raft_consensus.cc:348] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:24.134481 10584 raft_consensus.cc:374] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:24.134680 10584 raft_consensus.cc:729] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:24.135033 10584 consensus_queue.cc:260] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:24.135795 10584 ts_tablet_manager.cc:1398] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:10:24.136188 10584 tablet_bootstrap.cc:492] T cf05708a20a04c66875867f5453f7fb9 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:24.137465 10578 tablet_bootstrap.cc:492] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:24.137729 10578 ts_tablet_manager.cc:1367] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.155s user 0.005s sys 0.010s
I20240501 20:10:24.137873 7417 raft_consensus.cc:2932] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:24.139849 4232 consensus_queue.cc:237] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:24.140513 10582 tablet_bootstrap.cc:492] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:10:24.140635 10582 ts_tablet_manager.cc:1367] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.040s user 0.000s sys 0.005s
I20240501 20:10:24.141379 10582 raft_consensus.cc:348] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:24.141567 10582 raft_consensus.cc:374] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:24.141631 10582 raft_consensus.cc:729] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:10:24.142078 4498 raft_consensus.cc:1261] T e84a2462cb7e4e559fba887790a65a1f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:24.142685 9802 consensus_queue.cc:1035] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:24.141847 10582 consensus_queue.cc:260] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:24.144125 10580 tablet_bootstrap.cc:492] T accaff8513134f96b5502b5c5385d892 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:24.144328 10580 ts_tablet_manager.cc:1367] T accaff8513134f96b5502b5c5385d892 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.138s user 0.014s sys 0.024s
I20240501 20:10:24.145087 10580 raft_consensus.cc:348] T accaff8513134f96b5502b5c5385d892 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:24.145347 10580 raft_consensus.cc:374] T accaff8513134f96b5502b5c5385d892 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:24.145463 10582 ts_tablet_manager.cc:1398] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 20:10:24.145499 10580 raft_consensus.cc:729] T accaff8513134f96b5502b5c5385d892 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:24.144512 10587 tablet.cc:1722] T ecca8fc3a57047c396cd2551df962e5d P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:24.145807 10580 consensus_queue.cc:260] T accaff8513134f96b5502b5c5385d892 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:24.146147 10587 tablet.cc:1620] T ecca8fc3a57047c396cd2551df962e5d P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:24.148030 10580 ts_tablet_manager.cc:1398] T accaff8513134f96b5502b5c5385d892 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 20:10:24.138672 10578 raft_consensus.cc:348] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:24.151741 10578 raft_consensus.cc:374] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:24.151895 10578 raft_consensus.cc:729] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:24.152245 10578 consensus_queue.cc:260] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:24.155716 4088 catalog_manager.cc:5576] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447) does not have the latest schema for tablet ecca8fc3a57047c396cd2551df962e5d (table rebalancer_dynamic_table [id=198a355f97e3458f8a2c4889a3091902]). Expected version 2 got 1
I20240501 20:10:24.155793 10578 ts_tablet_manager.cc:1398] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20240501 20:10:24.156154 10578 tablet_bootstrap.cc:492] T 786296114e8a47c68df103934ec055d5 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:24.162427 10600 ts_tablet_manager.cc:902] T c147c98b674e499992f0b9f556ac6815 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Initiating tablet copy from peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:24.169201 10600 tablet_copy_client.cc:308] T c147c98b674e499992f0b9f556ac6815 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.70:36321
I20240501 20:10:24.170605 7175 tablet_copy_service.cc:140] P db0742ee3860468590eefef6ae6074fd: Received BeginTabletCopySession request for tablet c147c98b674e499992f0b9f556ac6815 from peer 1d34e18a1d394d3f91c9f50d1c7bea4e ({username='slave'} at 127.29.219.68:44315)
I20240501 20:10:24.170794 7175 tablet_copy_service.cc:161] P db0742ee3860468590eefef6ae6074fd: Beginning new tablet copy session on tablet c147c98b674e499992f0b9f556ac6815 from peer 1d34e18a1d394d3f91c9f50d1c7bea4e at {username='slave'} at 127.29.219.68:44315: session id = 1d34e18a1d394d3f91c9f50d1c7bea4e-c147c98b674e499992f0b9f556ac6815
I20240501 20:10:24.171175 10578 tablet_bootstrap.cc:654] T 786296114e8a47c68df103934ec055d5 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:24.171005 9802 raft_consensus.cc:2932] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:24.172055 10584 tablet_bootstrap.cc:654] T cf05708a20a04c66875867f5453f7fb9 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:24.173782 7175 tablet_copy_source_session.cc:205] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:24.174476 4498 raft_consensus.cc:2932] T e84a2462cb7e4e559fba887790a65a1f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:24.175136 10600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c147c98b674e499992f0b9f556ac6815. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:24.183633 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e84a2462cb7e4e559fba887790a65a1f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:24.186693 7419 raft_consensus.cc:1261] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:24.187408 9953 consensus_queue.cc:1035] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:24.189030 7419 raft_consensus.cc:2932] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:24.190287 10584 tablet_bootstrap.cc:492] T cf05708a20a04c66875867f5453f7fb9 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:24.190447 10584 ts_tablet_manager.cc:1367] T cf05708a20a04c66875867f5453f7fb9 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.054s user 0.000s sys 0.003s
I20240501 20:10:24.191248 10584 raft_consensus.cc:348] T cf05708a20a04c66875867f5453f7fb9 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:24.194531 10578 tablet_bootstrap.cc:492] T 786296114e8a47c68df103934ec055d5 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:24.195050 10578 ts_tablet_manager.cc:1367] T 786296114e8a47c68df103934ec055d5 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20240501 20:10:24.196007 10578 raft_consensus.cc:348] T 786296114e8a47c68df103934ec055d5 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:24.197105 10578 raft_consensus.cc:374] T 786296114e8a47c68df103934ec055d5 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:24.197381 10584 raft_consensus.cc:374] T cf05708a20a04c66875867f5453f7fb9 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:24.197539 10578 raft_consensus.cc:729] T 786296114e8a47c68df103934ec055d5 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:24.197885 10584 raft_consensus.cc:729] T cf05708a20a04c66875867f5453f7fb9 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:24.198220 10578 consensus_queue.cc:260] T 786296114e8a47c68df103934ec055d5 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:24.198803 10584 consensus_queue.cc:260] T cf05708a20a04c66875867f5453f7fb9 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:24.199782 10578 ts_tablet_manager.cc:1398] T 786296114e8a47c68df103934ec055d5 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 20:10:24.202093 10584 ts_tablet_manager.cc:1398] T cf05708a20a04c66875867f5453f7fb9 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.012s user 0.002s sys 0.001s
I20240501 20:10:24.202667 4498 raft_consensus.cc:1203] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:24.202942 10578 tablet_bootstrap.cc:492] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:24.207664 10338 raft_consensus.cc:1050] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d: attempting to promote NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b to VOTER
I20240501 20:10:24.207700 10578 tablet_bootstrap.cc:654] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:24.208474 10338 consensus_queue.cc:237] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:24.209748 4088 catalog_manager.cc:5484] T e84a2462cb7e4e559fba887790a65a1f P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:24.215180 7134 tablet_service.cc:1514] Processing DeleteTablet for tablet e84a2462cb7e4e559fba887790a65a1f with delete_type TABLET_DATA_TOMBSTONED (TS db0742ee3860468590eefef6ae6074fd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52036
I20240501 20:10:24.220312 4628 raft_consensus.cc:1261] T 7afb8e3154a64be78b8041b137b3681a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:24.221218 10368 consensus_queue.cc:1035] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:24.228528 7287 raft_consensus.cc:1261] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:24.229310 10368 consensus_queue.cc:1035] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:24.230427 10600 tablet_copy_client.cc:783] T c147c98b674e499992f0b9f556ac6815 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:24.230731 10600 tablet_copy_client.cc:647] T c147c98b674e499992f0b9f556ac6815 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:24.231132 4089 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:24.233489 10600 tablet_copy_client.cc:523] T c147c98b674e499992f0b9f556ac6815 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:24.236555 10579 tablet_bootstrap.cc:492] T accaff8513134f96b5502b5c5385d892 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:24.241439 10578 tablet_bootstrap.cc:492] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:24.242416 10579 ts_tablet_manager.cc:1367] T accaff8513134f96b5502b5c5385d892 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.145s user 0.007s sys 0.010s
I20240501 20:10:24.242698 10578 ts_tablet_manager.cc:1367] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.040s user 0.001s sys 0.003s
I20240501 20:10:24.244060 10575 tablet.cc:1722] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:24.244521 10579 raft_consensus.cc:348] T accaff8513134f96b5502b5c5385d892 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:24.245590 10576 tablet.cc:1722] T ecca8fc3a57047c396cd2551df962e5d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:24.249168 10609 tablet_replica.cc:330] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd: stopping tablet replica
I20240501 20:10:24.249518 10609 raft_consensus.cc:2229] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:24.250988 10609 raft_consensus.cc:2258] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:24.252081 10609 ts_tablet_manager.cc:1875] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:24.253595 7417 raft_consensus.cc:1203] T 7725fa9f98e9439797a532ed6f2f7dba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:10:24.257058 10579 raft_consensus.cc:374] T accaff8513134f96b5502b5c5385d892 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:24.258925 10600 tablet_bootstrap.cc:492] T c147c98b674e499992f0b9f556ac6815 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:24.259546 4087 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table_new_name [id=198a355f97e3458f8a2c4889a3091902] alter complete (version 3)
I20240501 20:10:24.260924 4498 raft_consensus.cc:1261] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:24.261072 10609 ts_tablet_manager.cc:1888] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:24.259227 10579 raft_consensus.cc:729] T accaff8513134f96b5502b5c5385d892 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:24.261556 10579 consensus_queue.cc:260] T accaff8513134f96b5502b5c5385d892 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:24.261798 10145 consensus_queue.cc:1035] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:24.266784 10609 log.cc:1196] T e84a2462cb7e4e559fba887790a65a1f P db0742ee3860468590eefef6ae6074fd: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/wals/e84a2462cb7e4e559fba887790a65a1f
I20240501 20:10:24.268010 4076 catalog_manager.cc:4830] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): tablet e84a2462cb7e4e559fba887790a65a1f (table rebalancer_extra_table_36 [id=2cb1d6b4f2204c5c93a6b7b02e2c8a05]) successfully deleted
I20240501 20:10:24.274596 10578 raft_consensus.cc:348] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:24.278314 10578 raft_consensus.cc:374] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:24.278508 10578 raft_consensus.cc:729] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:24.279186 10578 consensus_queue.cc:260] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:24.279846 10578 ts_tablet_manager.cc:1398] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20240501 20:10:24.288331 10579 ts_tablet_manager.cc:1398] T accaff8513134f96b5502b5c5385d892 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.045s user 0.001s sys 0.001s
I20240501 20:10:24.297704 10221 raft_consensus.cc:482] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:24.297951 10221 raft_consensus.cc:504] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:24.299037 10221 leader_election.cc:288] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:24.299875 7281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf05708a20a04c66875867f5453f7fb9" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:24.300175 7281 raft_consensus.cc:2454] T cf05708a20a04c66875867f5453f7fb9 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:10:24.300891 7354 leader_election.cc:302] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:24.301842 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf05708a20a04c66875867f5453f7fb9" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:10:24.302134 4365 raft_consensus.cc:2454] T cf05708a20a04c66875867f5453f7fb9 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:10:24.303421 10221 raft_consensus.cc:2781] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:24.303620 10221 raft_consensus.cc:482] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:24.303808 10221 raft_consensus.cc:3037] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:24.307673 10619 ts_tablet_manager.cc:902] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P 10b47782558c41ee91f5e164a372049a: Initiating tablet copy from peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:24.315975 10221 raft_consensus.cc:504] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:24.322786 10587 tablet.cc:1722] T ecca8fc3a57047c396cd2551df962e5d P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:24.328548 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf05708a20a04c66875867f5453f7fb9" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:10:24.328828 4365 raft_consensus.cc:3037] T cf05708a20a04c66875867f5453f7fb9 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:24.331547 10619 tablet_copy_client.cc:308] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P 10b47782558c41ee91f5e164a372049a: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.65:42941
I20240501 20:10:24.333370 4365 raft_consensus.cc:2454] T cf05708a20a04c66875867f5453f7fb9 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:24.336941 7355 leader_election.cc:302] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, b9957d4086bc4a76b18ed9265a5739d5; no voters:
I20240501 20:10:24.337545 10046 raft_consensus.cc:2781] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:24.337713 10046 raft_consensus.cc:686] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Becoming Leader. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Running, Role: LEADER
I20240501 20:10:24.338109 10046 consensus_queue.cc:237] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:24.353472 10600 tablet_bootstrap.cc:492] T c147c98b674e499992f0b9f556ac6815 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:24.353833 10600 tablet_bootstrap.cc:492] T c147c98b674e499992f0b9f556ac6815 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap complete.
I20240501 20:10:24.354080 10600 ts_tablet_manager.cc:1367] T c147c98b674e499992f0b9f556ac6815 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.095s user 0.005s sys 0.012s
I20240501 20:10:24.361687 4089 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:24.362035 4089 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:24.359694 10600 raft_consensus.cc:348] T c147c98b674e499992f0b9f556ac6815 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:24.364866 10600 raft_consensus.cc:729] T c147c98b674e499992f0b9f556ac6815 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: LEARNER
I20240501 20:10:24.365058 10600 consensus_queue.cc:260] T c147c98b674e499992f0b9f556ac6815 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:24.366079 4089 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet ecca8fc3a57047c396cd2551df962e5d
I20240501 20:10:24.367730 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet ecca8fc3a57047c396cd2551df962e5d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:24 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:24.367849 7400 tablet_service.cc:1514] Processing DeleteTablet for tablet ecca8fc3a57047c396cd2551df962e5d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:24 UTC) from {username='slave'} at 127.0.0.1:40182
I20240501 20:10:24.381552 10632 tablet_replica.cc:330] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5: stopping tablet replica
I20240501 20:10:24.381953 10632 raft_consensus.cc:2229] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:24.382335 10632 raft_consensus.cc:2258] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:24.383070 10466 raft_consensus.cc:1050] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e: attempting to promote NON_VOTER bceeae1c79964acabdae8ff771bf0c97 to VOTER
I20240501 20:10:24.383922 10466 consensus_queue.cc:237] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:24.385457 10600 ts_tablet_manager.cc:1398] T c147c98b674e499992f0b9f556ac6815 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.031s user 0.001s sys 0.003s
I20240501 20:10:24.387246 10632 ts_tablet_manager.cc:1875] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:24.390318 7175 tablet_copy_service.cc:342] P db0742ee3860468590eefef6ae6074fd: Request end of tablet copy session 1d34e18a1d394d3f91c9f50d1c7bea4e-c147c98b674e499992f0b9f556ac6815 received from {username='slave'} at 127.29.219.68:44315
I20240501 20:10:24.390462 7175 tablet_copy_service.cc:433] P db0742ee3860468590eefef6ae6074fd: ending tablet copy session 1d34e18a1d394d3f91c9f50d1c7bea4e-c147c98b674e499992f0b9f556ac6815 on tablet c147c98b674e499992f0b9f556ac6815 with peer 1d34e18a1d394d3f91c9f50d1c7bea4e
I20240501 20:10:24.400132 4254 tablet_copy_service.cc:140] P dca57823003e4729b769ca12367d7530: Received BeginTabletCopySession request for tablet 6cb1ca3dbf614f94b39e8ddd28ab04fc from peer 10b47782558c41ee91f5e164a372049a ({username='slave'} at 127.29.219.66:39201)
I20240501 20:10:24.400350 4254 tablet_copy_service.cc:161] P dca57823003e4729b769ca12367d7530: Beginning new tablet copy session on tablet 6cb1ca3dbf614f94b39e8ddd28ab04fc from peer 10b47782558c41ee91f5e164a372049a at {username='slave'} at 127.29.219.66:39201: session id = 10b47782558c41ee91f5e164a372049a-6cb1ca3dbf614f94b39e8ddd28ab04fc
I20240501 20:10:24.400810 7281 raft_consensus.cc:1261] T 41d6e6dce35e4d0da594f147a7f9bfc4 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:24.401473 10377 consensus_queue.cc:1035] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:24.404915 7155 raft_consensus.cc:1261] T 41d6e6dce35e4d0da594f147a7f9bfc4 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:24.405580 10627 consensus_queue.cc:1035] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:24.409590 4254 tablet_copy_source_session.cc:205] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:24.415123 4365 raft_consensus.cc:1261] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:24.415378 10632 ts_tablet_manager.cc:1888] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:24.415911 10627 consensus_queue.cc:1035] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:24.419231 10632 log.cc:1196] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/wals/ecca8fc3a57047c396cd2551df962e5d
I20240501 20:10:24.419778 10619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cb1ca3dbf614f94b39e8ddd28ab04fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:24.420380 10632 ts_tablet_manager.cc:1909] T ecca8fc3a57047c396cd2551df962e5d P b9957d4086bc4a76b18ed9265a5739d5: Deleting consensus metadata
I20240501 20:10:24.420958 9926 raft_consensus.cc:1050] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd: attempting to promote NON_VOTER 10b47782558c41ee91f5e164a372049a to VOTER
I20240501 20:10:24.421754 9926 consensus_queue.cc:237] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } }
I20240501 20:10:24.426054 10619 tablet_copy_client.cc:783] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P 10b47782558c41ee91f5e164a372049a: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:24.426272 10619 tablet_copy_client.cc:647] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P 10b47782558c41ee91f5e164a372049a: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:24.428444 4367 raft_consensus.cc:1261] T 030335599b754c81810fe0d139ef51f1 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:24.429191 10277 consensus_queue.cc:1035] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:24.429351 4079 catalog_manager.cc:4830] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): tablet ecca8fc3a57047c396cd2551df962e5d (table rebalancer_dynamic_table_new_name [id=198a355f97e3458f8a2c4889a3091902]) successfully deleted
I20240501 20:10:24.432631 7022 raft_consensus.cc:1261] T 030335599b754c81810fe0d139ef51f1 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:24.434907 10277 consensus_queue.cc:1035] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:24.438220 10145 raft_consensus.cc:482] T accaff8513134f96b5502b5c5385d892 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:24.438513 10145 raft_consensus.cc:504] T accaff8513134f96b5502b5c5385d892 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:24.439427 10145 leader_election.cc:288] T accaff8513134f96b5502b5c5385d892 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:24.444805 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "accaff8513134f96b5502b5c5385d892" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:24.445127 7419 raft_consensus.cc:2454] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:24.446012 6958 leader_election.cc:302] T accaff8513134f96b5502b5c5385d892 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:24.446534 7419 raft_consensus.cc:1261] T 030335599b754c81810fe0d139ef51f1 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:24.446554 10145 raft_consensus.cc:2781] T accaff8513134f96b5502b5c5385d892 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:24.446686 10145 raft_consensus.cc:482] T accaff8513134f96b5502b5c5385d892 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:24.446813 10145 raft_consensus.cc:3037] T accaff8513134f96b5502b5c5385d892 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:24.450342 10145 raft_consensus.cc:504] T accaff8513134f96b5502b5c5385d892 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:24.452940 10277 consensus_queue.cc:1035] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:24.458215 7417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "accaff8513134f96b5502b5c5385d892" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:24.458659 7417 raft_consensus.cc:3037] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:24.459132 10145 leader_election.cc:288] T accaff8513134f96b5502b5c5385d892 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Requested vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:24.466840 10391 raft_consensus.cc:482] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:24.467098 10391 raft_consensus.cc:504] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:24.468161 10391 leader_election.cc:288] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:24.469815 7421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786296114e8a47c68df103934ec055d5" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:24.470105 7421 raft_consensus.cc:2454] T 786296114e8a47c68df103934ec055d5 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:10:24.470639 4435 leader_election.cc:302] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, b9957d4086bc4a76b18ed9265a5739d5; no voters:
I20240501 20:10:24.471012 10391 raft_consensus.cc:2781] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:24.471114 10391 raft_consensus.cc:482] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:24.471191 10391 raft_consensus.cc:3037] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:24.474440 10391 raft_consensus.cc:504] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:24.476039 7421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786296114e8a47c68df103934ec055d5" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:24.476274 7421 raft_consensus.cc:3037] T 786296114e8a47c68df103934ec055d5 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:24.476291 10391 leader_election.cc:288] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Requested vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:24.479794 4089 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:24.481079 7421 raft_consensus.cc:2454] T 786296114e8a47c68df103934ec055d5 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:10:24.481940 4435 leader_election.cc:302] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, b9957d4086bc4a76b18ed9265a5739d5; no voters:
I20240501 20:10:24.482312 10230 raft_consensus.cc:2781] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:24.482471 10230 raft_consensus.cc:686] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:10:24.482807 10230 consensus_queue.cc:237] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:24.510016 10648 raft_consensus.cc:2932] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:24.528065 4232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "accaff8513134f96b5502b5c5385d892" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:24.528383 4232 raft_consensus.cc:2454] T accaff8513134f96b5502b5c5385d892 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:24.528337 4088 catalog_manager.cc:5484] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:24.530014 4232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "accaff8513134f96b5502b5c5385d892" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:10:24.530283 4232 raft_consensus.cc:3037] T accaff8513134f96b5502b5c5385d892 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:24.533052 10644 raft_consensus.cc:482] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:24.533234 10644 raft_consensus.cc:504] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:24.534459 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786296114e8a47c68df103934ec055d5" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:24.535544 7419 raft_consensus.cc:2379] T 786296114e8a47c68df103934ec055d5 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bceeae1c79964acabdae8ff771bf0c97 in current term 1: Already voted for candidate 3e974c03d3c546f9ab6c91e762a5426b in this term.
I20240501 20:10:24.536548 10644 leader_election.cc:288] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:24.539005 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786296114e8a47c68df103934ec055d5" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:24.539885 7222 leader_election.cc:302] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97; no voters: 3e974c03d3c546f9ab6c91e762a5426b, b9957d4086bc4a76b18ed9265a5739d5
I20240501 20:10:24.540270 10644 raft_consensus.cc:3037] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:24.543772 10644 raft_consensus.cc:2726] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:24.546257 4232 raft_consensus.cc:2454] T accaff8513134f96b5502b5c5385d892 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:24.547833 4367 raft_consensus.cc:2932] T 030335599b754c81810fe0d139ef51f1 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:24.552687 7022 raft_consensus.cc:2932] T 030335599b754c81810fe0d139ef51f1 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
W20240501 20:10:24.574858 10587 tablet_replica.cc:1402] Time spent flushing the Log queue took a long time: real 0.248s user 0.000s sys 0.000s
I20240501 20:10:24.578800 7002 tablet_service.cc:1467] Processing CreateTablet for tablet 66a950effa314a7782f0a0804db0615b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1d747f548b24460ba76f24e385fa5d81]), partition=
I20240501 20:10:24.578996 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 66a950effa314a7782f0a0804db0615b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1d747f548b24460ba76f24e385fa5d81]), partition=
I20240501 20:10:24.579226 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66a950effa314a7782f0a0804db0615b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:24.579373 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66a950effa314a7782f0a0804db0615b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:24.579667 7281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf05708a20a04c66875867f5453f7fb9" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:24.579883 7281 raft_consensus.cc:3037] T cf05708a20a04c66875867f5453f7fb9 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:24.584131 7281 raft_consensus.cc:2454] T cf05708a20a04c66875867f5453f7fb9 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:24.585542 10580 tablet_bootstrap.cc:492] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:24.586313 10579 tablet_bootstrap.cc:492] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:24.587144 10580 tablet_bootstrap.cc:654] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:24.588117 10579 tablet_bootstrap.cc:654] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:24.589977 10221 leader_election.cc:288] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:24.590168 10579 tablet_bootstrap.cc:492] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:24.590451 10579 ts_tablet_manager.cc:1367] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 20:10:24.591302 10579 raft_consensus.cc:348] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:24.591601 10579 raft_consensus.cc:374] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:24.591719 10579 raft_consensus.cc:729] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:24.592058 10579 consensus_queue.cc:260] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:24.592717 10579 ts_tablet_manager.cc:1398] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:24.595194 6957 leader_election.cc:302] T accaff8513134f96b5502b5c5385d892 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dca57823003e4729b769ca12367d7530, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:24.595481 10145 raft_consensus.cc:2781] T accaff8513134f96b5502b5c5385d892 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:24.595631 10145 raft_consensus.cc:686] T accaff8513134f96b5502b5c5385d892 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Becoming Leader. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Running, Role: LEADER
I20240501 20:10:24.595867 10145 consensus_queue.cc:237] T accaff8513134f96b5502b5c5385d892 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:24.603448 7419 raft_consensus.cc:2932] T 030335599b754c81810fe0d139ef51f1 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:24.607211 4088 catalog_manager.cc:5484] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:24.611415 10619 tablet_copy_client.cc:523] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P 10b47782558c41ee91f5e164a372049a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:24.611963 7154 consensus_queue.cc:237] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } }
I20240501 20:10:24.613865 7419 raft_consensus.cc:1261] T 030335599b754c81810fe0d139ef51f1 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:24.614491 9926 consensus_queue.cc:1035] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:24.615942 10619 tablet_bootstrap.cc:492] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:24.621089 7419 raft_consensus.cc:2932] T 030335599b754c81810fe0d139ef51f1 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:24.625574 10648 raft_consensus.cc:2932] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:24.644743 10619 tablet_bootstrap.cc:492] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P 10b47782558c41ee91f5e164a372049a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:24.649469 10619 tablet_bootstrap.cc:492] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P 10b47782558c41ee91f5e164a372049a: Bootstrap complete.
I20240501 20:10:24.650913 10619 ts_tablet_manager.cc:1367] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.002s
I20240501 20:10:24.655499 10619 raft_consensus.cc:348] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:24.661345 10619 raft_consensus.cc:729] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: LEARNER
I20240501 20:10:24.662004 10619 consensus_queue.cc:260] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:24.663923 10619 ts_tablet_manager.cc:1398] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240501 20:10:24.676720 10631 tablet_replica.cc:330] T ecca8fc3a57047c396cd2551df962e5d P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:24.677089 10631 raft_consensus.cc:2229] T ecca8fc3a57047c396cd2551df962e5d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:24.677384 10631 raft_consensus.cc:2258] T ecca8fc3a57047c396cd2551df962e5d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:24.678202 10629 ts_tablet_manager.cc:902] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530: Initiating tablet copy from peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:24.687348 10627 raft_consensus.cc:2932] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:24.691402 4364 raft_consensus.cc:2932] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:24.693794 7287 raft_consensus.cc:2932] T 41d6e6dce35e4d0da594f147a7f9bfc4 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:24.696812 4087 catalog_manager.cc:5484] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 7 to 8, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:24.704275 10466 raft_consensus.cc:1050] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e: attempting to promote NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 to VOTER
I20240501 20:10:24.704102 10631 ts_tablet_manager.cc:1875] T ecca8fc3a57047c396cd2551df962e5d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:24.705194 10466 consensus_queue.cc:237] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:24.710960 4076 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 030335599b754c81810fe0d139ef51f1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:24.714385 4091 catalog_manager.cc:5484] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:24.722951 10338 raft_consensus.cc:2932] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:24.736125 7002 tablet_service.cc:1514] Processing DeleteTablet for tablet 030335599b754c81810fe0d139ef51f1 with delete_type TABLET_DATA_TOMBSTONED (TS ef0a4ada548446b3a531cb39a3c4e38d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:24.737234 4496 raft_consensus.cc:2932] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:24.741227 10145 raft_consensus.cc:1050] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d: attempting to promote NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b to VOTER
I20240501 20:10:24.742063 10145 consensus_queue.cc:237] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:24.749717 4091 catalog_manager.cc:5484] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:24.757753 4631 raft_consensus.cc:1203] T c147c98b674e499992f0b9f556ac6815 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:24.773196 4496 raft_consensus.cc:1261] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:24.773885 10368 consensus_queue.cc:1035] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:24.777078 4631 raft_consensus.cc:1261] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:24.781888 10145 consensus_queue.cc:1035] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:24.795307 7421 raft_consensus.cc:1261] T 7725fa9f98e9439797a532ed6f2f7dba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:24.796361 10627 consensus_queue.cc:1035] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:24.797470 7154 raft_consensus.cc:1261] T 7725fa9f98e9439797a532ed6f2f7dba P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:24.798700 10377 consensus_queue.cc:1035] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:24.834661 10466 raft_consensus.cc:2932] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:24.840445 7421 raft_consensus.cc:2932] T 7725fa9f98e9439797a532ed6f2f7dba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:24.843400 4090 catalog_manager.cc:5484] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 7 to 8, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:24.850374 4627 consensus_queue.cc:237] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:24.852552 7421 raft_consensus.cc:1261] T 7725fa9f98e9439797a532ed6f2f7dba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:24.853189 10466 consensus_queue.cc:1035] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:24.858198 10377 raft_consensus.cc:2932] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:24.863219 7419 raft_consensus.cc:2932] T 7725fa9f98e9439797a532ed6f2f7dba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:24.806275 4628 raft_consensus.cc:2932] T 7afb8e3154a64be78b8041b137b3681a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
W20240501 20:10:24.868346 4169 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.69:46237, user_credentials={real_user=slave}} blocked reactor thread for 98761.9us
I20240501 20:10:24.871481 4079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7725fa9f98e9439797a532ed6f2f7dba with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:24.883487 7281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786296114e8a47c68df103934ec055d5" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:24.883857 7281 raft_consensus.cc:2454] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:10:24.887602 10631 ts_tablet_manager.cc:1888] T ecca8fc3a57047c396cd2551df962e5d P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:24.887779 7281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786296114e8a47c68df103934ec055d5" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:24.889091 10631 log.cc:1196] T ecca8fc3a57047c396cd2551df962e5d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/ecca8fc3a57047c396cd2551df962e5d
I20240501 20:10:24.889554 10631 ts_tablet_manager.cc:1909] T ecca8fc3a57047c396cd2551df962e5d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:24.890799 10046 consensus_queue.cc:1035] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:24.891607 7281 raft_consensus.cc:2454] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:10:24.891714 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet ecca8fc3a57047c396cd2551df962e5d (table rebalancer_dynamic_table_new_name [id=198a355f97e3458f8a2c4889a3091902]) successfully deleted
I20240501 20:10:24.892874 7417 raft_consensus.cc:2454] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:24.900808 4363 raft_consensus.cc:1261] T 030335599b754c81810fe0d139ef51f1 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:24.904757 10252 consensus_queue.cc:1035] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:24.904459 7283 raft_consensus.cc:2932] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:24.908365 4363 raft_consensus.cc:2932] T 030335599b754c81810fe0d139ef51f1 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:24.912961 10046 consensus_queue.cc:1035] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:24.936175 4087 catalog_manager.cc:5484] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72). New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:24.941881 10391 consensus_queue.cc:1035] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:24.846712 7154 raft_consensus.cc:2932] T 7725fa9f98e9439797a532ed6f2f7dba P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:24.972208 10580 tablet_bootstrap.cc:492] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:24.972486 10580 ts_tablet_manager.cc:1367] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.387s user 0.033s sys 0.089s
I20240501 20:10:24.973580 10580 raft_consensus.cc:348] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:24.973946 10580 raft_consensus.cc:374] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:24.974084 10580 raft_consensus.cc:729] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:24.974802 10580 consensus_queue.cc:260] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:24.975560 10580 ts_tablet_manager.cc:1398] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:10:24.977854 7267 tablet_service.cc:1514] Processing DeleteTablet for tablet ecca8fc3a57047c396cd2551df962e5d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:24 UTC) from {username='slave'} at 127.0.0.1:54166
I20240501 20:10:24.978211 7266 tablet_service.cc:1467] Processing CreateTablet for tablet 66a950effa314a7782f0a0804db0615b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1d747f548b24460ba76f24e385fa5d81]), partition=
I20240501 20:10:24.978740 7266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66a950effa314a7782f0a0804db0615b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:24.987591 7281 raft_consensus.cc:1261] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:24.991191 10662 tablet_replica.cc:330] T 030335599b754c81810fe0d139ef51f1 P ef0a4ada548446b3a531cb39a3c4e38d: stopping tablet replica
I20240501 20:10:24.991523 10662 raft_consensus.cc:2229] T 030335599b754c81810fe0d139ef51f1 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:24.991853 10662 raft_consensus.cc:2258] T 030335599b754c81810fe0d139ef51f1 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:24.993340 10662 ts_tablet_manager.cc:1875] T 030335599b754c81810fe0d139ef51f1 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:25.001497 10684 tablet_bootstrap.cc:492] T 66a950effa314a7782f0a0804db0615b P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:25.003942 10684 tablet_bootstrap.cc:654] T 66a950effa314a7782f0a0804db0615b P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
W20240501 20:10:25.010692 7155 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.603s user 0.036s sys 0.045s
W20240501 20:10:25.010932 7155 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.603s user 0.036s sys 0.045s
I20240501 20:10:25.013208 7155 raft_consensus.cc:2932] T 41d6e6dce35e4d0da594f147a7f9bfc4 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:25.021684 10368 raft_consensus.cc:2932] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:25.026151 4496 raft_consensus.cc:2932] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:25.031630 10368 consensus_queue.cc:1035] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:25.048578 4627 raft_consensus.cc:2932] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:25.056084 10683 tablet_replica.cc:330] T ecca8fc3a57047c396cd2551df962e5d P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:25.056322 10683 raft_consensus.cc:2229] T ecca8fc3a57047c396cd2551df962e5d P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:25.056712 10683 raft_consensus.cc:2258] T ecca8fc3a57047c396cd2551df962e5d P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:25.059063 10683 ts_tablet_manager.cc:1875] T ecca8fc3a57047c396cd2551df962e5d P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:25.059219 10391 consensus_queue.cc:1035] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:25.061575 10662 ts_tablet_manager.cc:1888] T 030335599b754c81810fe0d139ef51f1 P ef0a4ada548446b3a531cb39a3c4e38d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:25.061776 10662 log.cc:1196] T 030335599b754c81810fe0d139ef51f1 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/wals/030335599b754c81810fe0d139ef51f1
I20240501 20:10:24.946066 4088 catalog_manager.cc:5484] T accaff8513134f96b5502b5c5385d892 P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69). New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:25.033632 4091 catalog_manager.cc:5484] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:25.066237 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet 030335599b754c81810fe0d139ef51f1 (table rebalancer_extra_table_24 [id=b446a76fc1184ee88b0b983584d3f5f4]) successfully deleted
I20240501 20:10:25.085501 4254 tablet_copy_service.cc:342] P dca57823003e4729b769ca12367d7530: Request end of tablet copy session 10b47782558c41ee91f5e164a372049a-6cb1ca3dbf614f94b39e8ddd28ab04fc received from {username='slave'} at 127.29.219.66:39201
I20240501 20:10:25.085667 4254 tablet_copy_service.cc:433] P dca57823003e4729b769ca12367d7530: ending tablet copy session 10b47782558c41ee91f5e164a372049a-6cb1ca3dbf614f94b39e8ddd28ab04fc on tablet 6cb1ca3dbf614f94b39e8ddd28ab04fc with peer 10b47782558c41ee91f5e164a372049a
I20240501 20:10:25.089066 10610 consensus_queue.cc:1035] T accaff8513134f96b5502b5c5385d892 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:25.095513 7019 consensus_queue.cc:237] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:25.100626 4496 raft_consensus.cc:1261] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:25.101545 10338 consensus_queue.cc:1035] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:25.106207 7019 consensus_queue.cc:237] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:25.108095 10672 raft_consensus.cc:482] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:25.108296 10672 raft_consensus.cc:504] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:25.108814 4498 raft_consensus.cc:1261] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:25.109457 10672 leader_election.cc:288] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:25.109591 10338 consensus_queue.cc:1035] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:24.943397 4090 catalog_manager.cc:5484] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 8 to 9, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:25.112613 10145 raft_consensus.cc:2932] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:25.113255 4498 raft_consensus.cc:2932] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:25.119143 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7afb8e3154a64be78b8041b137b3681a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:25.121271 4084 catalog_manager.cc:5484] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: config changed from index 3 to 4, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:25.126302 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet 7afb8e3154a64be78b8041b137b3681a with delete_type TABLET_DATA_TOMBSTONED (TS 1d34e18a1d394d3f91c9f50d1c7bea4e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:25.126703 10631 tablet_replica.cc:330] T 7afb8e3154a64be78b8041b137b3681a P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:25.126915 10631 raft_consensus.cc:2229] T 7afb8e3154a64be78b8041b137b3681a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:25.127166 10631 raft_consensus.cc:2258] T 7afb8e3154a64be78b8041b137b3681a P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:25.128137 10631 ts_tablet_manager.cc:1875] T 7afb8e3154a64be78b8041b137b3681a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:25.135319 10683 ts_tablet_manager.cc:1888] T ecca8fc3a57047c396cd2551df962e5d P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:25.135550 10683 log.cc:1196] T ecca8fc3a57047c396cd2551df962e5d P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/ecca8fc3a57047c396cd2551df962e5d
W20240501 20:10:25.135883 7220 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.790s user 0.040s sys 0.060s
W20240501 20:10:25.136196 7220 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.790s user 0.041s sys 0.060s
W20240501 20:10:25.136673 7220 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.70:36321, user_credentials={real_user=slave}} blocked reactor thread for 790745us
I20240501 20:10:25.141450 10631 ts_tablet_manager.cc:1888] T 7afb8e3154a64be78b8041b137b3681a P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:25.141638 10631 log.cc:1196] T 7afb8e3154a64be78b8041b137b3681a P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/7afb8e3154a64be78b8041b137b3681a
I20240501 20:10:25.142604 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet 7afb8e3154a64be78b8041b137b3681a (table rebalancer_extra_table_24 [id=b446a76fc1184ee88b0b983584d3f5f4]) successfully deleted
I20240501 20:10:25.148082 10683 ts_tablet_manager.cc:1909] T ecca8fc3a57047c396cd2551df962e5d P bceeae1c79964acabdae8ff771bf0c97: Deleting consensus metadata
I20240501 20:10:25.160635 10684 tablet_bootstrap.cc:492] T 66a950effa314a7782f0a0804db0615b P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:25.167272 10684 ts_tablet_manager.cc:1367] T 66a950effa314a7782f0a0804db0615b P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.162s user 0.007s sys 0.000s
I20240501 20:10:25.167423 10629 tablet_copy_client.cc:308] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.69:46237
I20240501 20:10:25.166713 7283 raft_consensus.cc:2932] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:25.168227 10684 raft_consensus.cc:348] T 66a950effa314a7782f0a0804db0615b P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:25.168532 10684 raft_consensus.cc:374] T 66a950effa314a7782f0a0804db0615b P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:25.169029 10684 raft_consensus.cc:729] T 66a950effa314a7782f0a0804db0615b P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:25.169302 10684 consensus_queue.cc:260] T 66a950effa314a7782f0a0804db0615b P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:25.172398 7153 raft_consensus.cc:1261] T 7725fa9f98e9439797a532ed6f2f7dba P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:25.173419 4232 raft_consensus.cc:1261] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:25.173926 10382 consensus_queue.cc:1035] T 7725fa9f98e9439797a532ed6f2f7dba P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:25.174233 10684 ts_tablet_manager.cc:1398] T 66a950effa314a7782f0a0804db0615b P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:10:25.176270 4365 raft_consensus.cc:1203] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:25.177687 7154 raft_consensus.cc:2932] T 7725fa9f98e9439797a532ed6f2f7dba P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
W20240501 20:10:25.186524 6957 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.67:34595, user_credentials={real_user=slave}} blocked reactor thread for 57140.2us
I20240501 20:10:25.187798 10610 raft_consensus.cc:2932] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:25.193852 4496 raft_consensus.cc:2932] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:25.199715 10252 raft_consensus.cc:1050] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd: attempting to promote NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e to VOTER
I20240501 20:10:25.200590 10252 consensus_queue.cc:237] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } }
I20240501 20:10:25.211197 7283 raft_consensus.cc:1261] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:25.211259 4628 raft_consensus.cc:1261] T c147c98b674e499992f0b9f556ac6815 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:25.212139 10252 consensus_queue.cc:1035] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:25.212059 4087 catalog_manager.cc:5484] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:25.212617 9926 consensus_queue.cc:1035] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:25.215996 7019 raft_consensus.cc:1261] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:25.216337 7281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66a950effa314a7782f0a0804db0615b" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:25.216601 7281 raft_consensus.cc:2454] T 66a950effa314a7782f0a0804db0615b P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:25.217845 10252 consensus_queue.cc:1035] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:25.218297 7281 raft_consensus.cc:1261] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:25.218992 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet ecca8fc3a57047c396cd2551df962e5d (table rebalancer_dynamic_table_new_name [id=198a355f97e3458f8a2c4889a3091902]) successfully deleted
I20240501 20:10:25.235147 7267 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f63c61d85524d87a4b911c321f486c4 with delete_type TABLET_DATA_TOMBSTONED (TS bceeae1c79964acabdae8ff771bf0c97 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54166
I20240501 20:10:25.235531 10683 tablet_replica.cc:330] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:25.235775 10683 raft_consensus.cc:2229] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:25.235973 10683 raft_consensus.cc:2258] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:25.237143 10683 ts_tablet_manager.cc:1875] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:25.246451 10683 ts_tablet_manager.cc:1888] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:25.246727 10683 log.cc:1196] T 3f63c61d85524d87a4b911c321f486c4 P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/3f63c61d85524d87a4b911c321f486c4
I20240501 20:10:25.247862 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet 3f63c61d85524d87a4b911c321f486c4 (table rebalancer_extra_table_12 [id=817a9d24b5a445c4be8c44bcb31c8e41]) successfully deleted
W20240501 20:10:25.256843 6958 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.72:39817, user_credentials={real_user=slave}} blocked reactor thread for 134135us
I20240501 20:10:25.258386 4168 leader_election.cc:302] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:25.262099 10368 raft_consensus.cc:482] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:25.262459 10368 raft_consensus.cc:504] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:25.263973 10368 leader_election.cc:288] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:25.294976 10672 raft_consensus.cc:2781] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:25.295181 10672 raft_consensus.cc:482] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:25.295305 10672 raft_consensus.cc:3037] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:25.298010 10672 raft_consensus.cc:504] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:25.298535 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet 7725fa9f98e9439797a532ed6f2f7dba with delete_type TABLET_DATA_TOMBSTONED (TS dca57823003e4729b769ca12367d7530 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:25.298835 10672 leader_election.cc:288] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:25.303607 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66a950effa314a7782f0a0804db0615b" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:25.303886 7287 raft_consensus.cc:3037] T 66a950effa314a7782f0a0804db0615b P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:25.309334 7287 raft_consensus.cc:2454] T 66a950effa314a7782f0a0804db0615b P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:25.318110 4168 leader_election.cc:302] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:25.320549 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66a950effa314a7782f0a0804db0615b" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:25.320852 7022 raft_consensus.cc:2454] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:25.323241 10672 raft_consensus.cc:2781] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:25.323428 10672 raft_consensus.cc:686] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:10:25.323792 10672 consensus_queue.cc:237] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:25.325860 4627 raft_consensus.cc:1261] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:25.325703 10252 raft_consensus.cc:2932] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:25.326875 4627 raft_consensus.cc:2932] T c147c98b674e499992f0b9f556ac6815 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:25.332355 7287 raft_consensus.cc:2932] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:25.336442 4091 catalog_manager.cc:5484] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:25.343341 7154 consensus_queue.cc:237] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } }
I20240501 20:10:25.343681 7042 tablet_copy_service.cc:140] P ef0a4ada548446b3a531cb39a3c4e38d: Received BeginTabletCopySession request for tablet 743d7fd2ae014352a5681c2a8ee4447d from peer dca57823003e4729b769ca12367d7530 ({username='slave'} at 127.29.219.65:40207)
I20240501 20:10:25.343920 7042 tablet_copy_service.cc:161] P ef0a4ada548446b3a531cb39a3c4e38d: Beginning new tablet copy session on tablet 743d7fd2ae014352a5681c2a8ee4447d from peer dca57823003e4729b769ca12367d7530 at {username='slave'} at 127.29.219.65:40207: session id = dca57823003e4729b769ca12367d7530-743d7fd2ae014352a5681c2a8ee4447d
I20240501 20:10:25.346066 7287 raft_consensus.cc:1261] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:25.346562 10696 consensus_queue.cc:1035] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:25.354820 10252 raft_consensus.cc:2932] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:25.361079 10710 tablet_replica.cc:330] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:25.361886 10710 raft_consensus.cc:2229] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:25.362375 10710 raft_consensus.cc:2258] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:25.363394 10710 ts_tablet_manager.cc:1875] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:25.363902 7281 raft_consensus.cc:2932] T c147c98b674e499992f0b9f556ac6815 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:25.374161 7042 tablet_copy_source_session.cc:205] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:25.375793 4628 raft_consensus.cc:1261] T c147c98b674e499992f0b9f556ac6815 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:25.376413 10648 consensus_queue.cc:1035] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:25.378201 4076 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c147c98b674e499992f0b9f556ac6815 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:25.379391 10710 ts_tablet_manager.cc:1888] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 20:10:25.379526 10710 log.cc:1196] T 7725fa9f98e9439797a532ed6f2f7dba P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/7725fa9f98e9439797a532ed6f2f7dba
I20240501 20:10:25.381752 4084 catalog_manager.cc:5484] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:25.397976 4628 raft_consensus.cc:2932] T c147c98b674e499992f0b9f556ac6815 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:25.395364 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet 7725fa9f98e9439797a532ed6f2f7dba (table rebalance_test_table_1 [id=eccfbde58fa5455b9f89a20ad3a37f39]) successfully deleted
I20240501 20:10:25.403718 4091 catalog_manager.cc:5484] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:25.439954 10338 consensus_queue.cc:1035] T 3f63c61d85524d87a4b911c321f486c4 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:25.458272 4232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66a950effa314a7782f0a0804db0615b" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
W20240501 20:10:25.461159 6956 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.71:39447, user_credentials={real_user=slave}} blocked reactor thread for 274500us
I20240501 20:10:25.462512 4627 consensus_queue.cc:237] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:25.468572 7153 raft_consensus.cc:1261] T 41d6e6dce35e4d0da594f147a7f9bfc4 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:25.468890 7281 raft_consensus.cc:1261] T 41d6e6dce35e4d0da594f147a7f9bfc4 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:25.470031 10377 consensus_queue.cc:1035] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:25.472435 10382 consensus_queue.cc:1035] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:25.479944 10382 raft_consensus.cc:2932] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:25.483482 7287 raft_consensus.cc:2932] T 41d6e6dce35e4d0da594f147a7f9bfc4 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:25.474597 4627 raft_consensus.cc:2932] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:25.509743 4091 catalog_manager.cc:5484] T 41d6e6dce35e4d0da594f147a7f9bfc4 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index 8 to 9, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:25.516480 4346 tablet_service.cc:1514] Processing DeleteTablet for tablet 41d6e6dce35e4d0da594f147a7f9bfc4 with delete_type TABLET_DATA_TOMBSTONED (TS 10b47782558c41ee91f5e164a372049a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:25.520370 10701 consensus_queue.cc:1035] T accaff8513134f96b5502b5c5385d892 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:25.520430 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66a950effa314a7782f0a0804db0615b" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:25.520733 7019 raft_consensus.cc:3037] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:25.524614 7019 raft_consensus.cc:2454] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:25.538498 7153 raft_consensus.cc:2932] T 41d6e6dce35e4d0da594f147a7f9bfc4 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:25.546551 4084 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:25.571115 4079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 41d6e6dce35e4d0da594f147a7f9bfc4 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:25.583220 10629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 743d7fd2ae014352a5681c2a8ee4447d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:25.584239 7281 raft_consensus.cc:1261] T 66a950effa314a7782f0a0804db0615b P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:25.585041 10714 consensus_queue.cc:1035] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:25.613634 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3f63c61d85524d87a4b911c321f486c4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:25.614776 4091 catalog_manager.cc:5576] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447) does not have the latest schema for tablet 66a950effa314a7782f0a0804db0615b (table rebalancer_dynamic_table [id=1d747f548b24460ba76f24e385fa5d81]). Expected version 1 got 0
I20240501 20:10:25.635739 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66a950effa314a7782f0a0804db0615b" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:25.640208 7002 tablet_service.cc:1514] Processing DeleteTablet for tablet c147c98b674e499992f0b9f556ac6815 with delete_type TABLET_DATA_TOMBSTONED (TS ef0a4ada548446b3a531cb39a3c4e38d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:25.641769 6956 leader_election.cc:302] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ef0a4ada548446b3a531cb39a3c4e38d; no voters: bceeae1c79964acabdae8ff771bf0c97, dca57823003e4729b769ca12367d7530
I20240501 20:10:25.642230 10703 consensus_queue.cc:1035] T 7afb8e3154a64be78b8041b137b3681a P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:25.642443 10338 raft_consensus.cc:2726] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:25.645699 10722 tablet_replica.cc:330] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d: stopping tablet replica
I20240501 20:10:25.646477 10722 raft_consensus.cc:2229] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:25.654187 10722 pending_rounds.cc:62] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d: Trying to abort 1 pending ops.
I20240501 20:10:25.654352 10722 pending_rounds.cc:68] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7022977946419154944 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "c147c98b674e499992f0b9f556ac6815" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } } }
I20240501 20:10:25.655006 10722 raft_consensus.cc:2894] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240501 20:10:25.655136 7019 raft_consensus.cc:1261] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:25.655407 10722 raft_consensus.cc:2866] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:10:25.655565 10722 raft_consensus.cc:2258] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:25.657423 10722 ts_tablet_manager.cc:1875] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:25.660961 10350 consensus_queue.cc:1035] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:25.674026 10722 ts_tablet_manager.cc:1888] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:25.676810 10722 log.cc:1196] T c147c98b674e499992f0b9f556ac6815 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/wals/c147c98b674e499992f0b9f556ac6815
I20240501 20:10:25.678809 4091 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:25.694250 4477 tablet_service.cc:1467] Processing CreateTablet for tablet 9e0ecc1043af4bc1ab31dfa80ea47bb4 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4a624bfe1b6b4d6b8ab2c2c578c11688]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:25.694774 4477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e0ecc1043af4bc1ab31dfa80ea47bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:25.695727 7134 tablet_service.cc:1467] Processing CreateTablet for tablet b67250c5773f4c9d9254d0306158ccde (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4a624bfe1b6b4d6b8ab2c2c578c11688]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:25.696152 7134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b67250c5773f4c9d9254d0306158ccde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:25.699155 7400 tablet_service.cc:1467] Processing CreateTablet for tablet 9e0ecc1043af4bc1ab31dfa80ea47bb4 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4a624bfe1b6b4d6b8ab2c2c578c11688]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:25.699566 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e0ecc1043af4bc1ab31dfa80ea47bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:25.703776 7266 tablet_service.cc:1467] Processing CreateTablet for tablet b67250c5773f4c9d9254d0306158ccde (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4a624bfe1b6b4d6b8ab2c2c578c11688]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:25.704201 7267 tablet_service.cc:1467] Processing CreateTablet for tablet 9e0ecc1043af4bc1ab31dfa80ea47bb4 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4a624bfe1b6b4d6b8ab2c2c578c11688]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:25.704977 7265 tablet_service.cc:1467] Processing CreateTablet for tablet aecca0e22b084a8b95c98ef7e7b90b29 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4a624bfe1b6b4d6b8ab2c2c578c11688]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:25.704236 7266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b67250c5773f4c9d9254d0306158ccde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:25.706210 7265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aecca0e22b084a8b95c98ef7e7b90b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:25.706144 4478 tablet_service.cc:1467] Processing CreateTablet for tablet aecca0e22b084a8b95c98ef7e7b90b29 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4a624bfe1b6b4d6b8ab2c2c578c11688]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:25.706588 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aecca0e22b084a8b95c98ef7e7b90b29. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 20:10:25.706858 6956 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.219.71:39447, user_credentials={real_user=slave}} blocked reactor thread for 65135.7us
I20240501 20:10:25.708628 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet c147c98b674e499992f0b9f556ac6815 (table rebalancer_extra_table_16 [id=514d975194074c92815ac14399279d97]) successfully deleted
I20240501 20:10:25.710198 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e0ecc1043af4bc1ab31dfa80ea47bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:25.711601 7287 raft_consensus.cc:2932] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:25.724699 10721 tablet.cc:1722] T 66a950effa314a7782f0a0804db0615b P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:25.725369 10721 tablet.cc:1620] T 66a950effa314a7782f0a0804db0615b P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:25.726591 10720 tablet.cc:1722] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:25.727344 10720 tablet.cc:1620] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530: MemRowSet was empty: no flush needed.
I20240501 20:10:25.745286 7002 tablet_service.cc:1467] Processing CreateTablet for tablet b67250c5773f4c9d9254d0306158ccde (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4a624bfe1b6b4d6b8ab2c2c578c11688]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:25.745820 7002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b67250c5773f4c9d9254d0306158ccde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:25.748420 10714 raft_consensus.cc:1050] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530: attempting to promote NON_VOTER 10b47782558c41ee91f5e164a372049a to VOTER
I20240501 20:10:25.745287 7001 tablet_service.cc:1467] Processing CreateTablet for tablet aecca0e22b084a8b95c98ef7e7b90b29 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=4a624bfe1b6b4d6b8ab2c2c578c11688]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:25.753177 7001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aecca0e22b084a8b95c98ef7e7b90b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:25.757256 10714 consensus_queue.cc:237] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } }
I20240501 20:10:25.764137 10725 tablet_bootstrap.cc:492] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:25.766387 10725 tablet_bootstrap.cc:654] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:25.769039 10729 tablet_bootstrap.cc:492] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:25.770085 4091 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=1d747f548b24460ba76f24e385fa5d81] alter complete (version 1)
I20240501 20:10:25.771420 10729 tablet_bootstrap.cc:654] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:25.772776 10731 tablet.cc:1722] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:25.773517 10731 tablet.cc:1620] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
W20240501 20:10:25.774981 4301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.65:42941, user_credentials={real_user=slave}} blocked reactor thread for 194586us
I20240501 20:10:25.776651 4091 catalog_manager.cc:5576] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237) does not have the latest schema for tablet 66a950effa314a7782f0a0804db0615b (table rebalancer_dynamic_table [id=1d747f548b24460ba76f24e385fa5d81]). Expected version 1 got 0
I20240501 20:10:25.779315 10726 tablet_bootstrap.cc:492] T aecca0e22b084a8b95c98ef7e7b90b29 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:25.781003 10726 tablet_bootstrap.cc:654] T aecca0e22b084a8b95c98ef7e7b90b29 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:25.789625 10733 tablet_bootstrap.cc:492] T aecca0e22b084a8b95c98ef7e7b90b29 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:25.792559 4084 catalog_manager.cc:5576] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447) does not have the latest schema for tablet 66a950effa314a7782f0a0804db0615b (table rebalancer_dynamic_table [id=1d747f548b24460ba76f24e385fa5d81]). Expected version 1 got 0
I20240501 20:10:25.792784 10725 tablet_bootstrap.cc:492] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:25.792960 7153 raft_consensus.cc:1261] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:25.793587 10673 consensus_queue.cc:1035] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:25.794361 10729 tablet_bootstrap.cc:492] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:25.794569 10729 ts_tablet_manager.cc:1367] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240501 20:10:25.795620 10729 raft_consensus.cc:348] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:25.795989 10729 raft_consensus.cc:374] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:25.796137 10729 raft_consensus.cc:729] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:25.796450 10729 consensus_queue.cc:260] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:25.799959 10725 ts_tablet_manager.cc:1367] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.004s
I20240501 20:10:25.800829 10725 raft_consensus.cc:348] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:25.801106 10725 raft_consensus.cc:374] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:25.801226 10725 raft_consensus.cc:729] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:25.801638 10725 consensus_queue.cc:260] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:25.802569 10725 ts_tablet_manager.cc:1398] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:10:25.803299 10725 tablet_bootstrap.cc:492] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:25.804962 10729 ts_tablet_manager.cc:1398] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 20:10:25.806738 7281 raft_consensus.cc:1261] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:25.807673 10725 tablet_bootstrap.cc:654] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:25.807555 10673 consensus_queue.cc:1035] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:25.809482 10726 tablet_bootstrap.cc:492] T aecca0e22b084a8b95c98ef7e7b90b29 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:10:25.810081 10726 ts_tablet_manager.cc:1367] T aecca0e22b084a8b95c98ef7e7b90b29 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20240501 20:10:25.810942 10726 raft_consensus.cc:348] T aecca0e22b084a8b95c98ef7e7b90b29 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:25.811291 10726 raft_consensus.cc:374] T aecca0e22b084a8b95c98ef7e7b90b29 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:25.811381 10726 raft_consensus.cc:729] T aecca0e22b084a8b95c98ef7e7b90b29 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:10:25.811622 10726 consensus_queue.cc:260] T aecca0e22b084a8b95c98ef7e7b90b29 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:25.812131 10726 ts_tablet_manager.cc:1398] T aecca0e22b084a8b95c98ef7e7b90b29 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:25.812534 10726 tablet_bootstrap.cc:492] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:25.814373 10726 tablet_bootstrap.cc:654] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:25.820808 10727 tablet_bootstrap.cc:492] T b67250c5773f4c9d9254d0306158ccde P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:25.822798 10727 tablet_bootstrap.cc:654] T b67250c5773f4c9d9254d0306158ccde P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
W20240501 20:10:25.824831 7355 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.67:34595, user_credentials={real_user=slave}} blocked reactor thread for 130497us
I20240501 20:10:25.831406 10726 tablet_bootstrap.cc:492] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:10:25.831568 10726 ts_tablet_manager.cc:1367] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20240501 20:10:25.832331 10726 raft_consensus.cc:348] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:25.832598 10726 raft_consensus.cc:374] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:25.832702 10726 raft_consensus.cc:729] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:10:25.833034 10726 consensus_queue.cc:260] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:25.833727 10726 ts_tablet_manager.cc:1398] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:25.847838 4365 raft_consensus.cc:1261] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:25.849576 10727 tablet_bootstrap.cc:492] T b67250c5773f4c9d9254d0306158ccde P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:25.849895 10727 ts_tablet_manager.cc:1367] T b67250c5773f4c9d9254d0306158ccde P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.004s
I20240501 20:10:25.851167 10727 raft_consensus.cc:348] T b67250c5773f4c9d9254d0306158ccde P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:25.851573 10727 raft_consensus.cc:374] T b67250c5773f4c9d9254d0306158ccde P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:25.853964 10727 raft_consensus.cc:729] T b67250c5773f4c9d9254d0306158ccde P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:25.854471 10727 consensus_queue.cc:260] T b67250c5773f4c9d9254d0306158ccde P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:25.855587 10727 ts_tablet_manager.cc:1398] T b67250c5773f4c9d9254d0306158ccde P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:10:25.878505 10715 tablet_replica.cc:330] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:25.878851 10715 raft_consensus.cc:2229] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:25.879186 10715 raft_consensus.cc:2258] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:25.882174 10715 ts_tablet_manager.cc:1875] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:25.883569 7287 raft_consensus.cc:2932] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:25.888967 10649 raft_consensus.cc:482] T aecca0e22b084a8b95c98ef7e7b90b29 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:25.889165 10649 raft_consensus.cc:504] T aecca0e22b084a8b95c98ef7e7b90b29 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:25.889968 10649 leader_election.cc:288] T aecca0e22b084a8b95c98ef7e7b90b29 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:25.893111 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecca0e22b084a8b95c98ef7e7b90b29" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
W20240501 20:10:25.893839 4434 leader_election.cc:341] T aecca0e22b084a8b95c98ef7e7b90b29 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:25.894563 4091 catalog_manager.cc:5484] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:25.877841 10673 raft_consensus.cc:2932] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:25.895255 7281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecca0e22b084a8b95c98ef7e7b90b29" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
W20240501 20:10:25.902177 4433 leader_election.cc:341] T aecca0e22b084a8b95c98ef7e7b90b29 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:25.902352 4433 leader_election.cc:302] T aecca0e22b084a8b95c98ef7e7b90b29 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b; no voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:25.905117 10645 raft_consensus.cc:2726] T aecca0e22b084a8b95c98ef7e7b90b29 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:25.906251 10671 consensus_queue.cc:1035] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:25.910677 7153 raft_consensus.cc:2932] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:25.913406 10629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.29.219.69:51289 (request call id 90) took 1558 ms. Trace:
I20240501 20:10:25.913792 10629 rpcz_store.cc:276] 0501 20:10:24.354979 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 20:10:24.355185 (+ 206us) service_pool.cc:225] Handling call
0501 20:10:24.678338 (+323153us) ts_tablet_manager.cc:903] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530: Initiating tablet copy from peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
0501 20:10:25.582704 (+904366us) tablet_copy_client.cc:353] Tablet copy session begun
0501 20:10:25.700864 (+118160us) tablet_metadata.cc:641] Metadata flushed
0501 20:10:25.701011 (+ 147us) tablet_copy_client.cc:482] Wrote new tablet metadata
0501 20:10:25.872627 (+171616us) tablet_copy_client.cc:403] Wrote new consensus metadata
0501 20:10:25.875950 (+ 3323us) tablet_replica.cc:205] Creating consensus instance
0501 20:10:25.913353 (+ 37403us) ts_tablet_manager.cc:958] Replica registered.
0501 20:10:25.913378 (+ 25us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":114,"tablet-copy.queue_time_us":322782,"thread_start_us":801398,"threads_started":5}
I20240501 20:10:25.916435 10629 tablet_copy_client.cc:783] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:25.917073 10629 tablet_copy_client.cc:647] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:25.939224 4087 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:25.941164 10715 ts_tablet_manager.cc:1888] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:10:25.944890 10715 log.cc:1196] T 41d6e6dce35e4d0da594f147a7f9bfc4 P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/41d6e6dce35e4d0da594f147a7f9bfc4
I20240501 20:10:25.948084 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet 41d6e6dce35e4d0da594f147a7f9bfc4 (table rebalance_test_table_5 [id=1e137c7380c64655a3b011211c536d72]) successfully deleted
I20240501 20:10:25.954198 10724 raft_consensus.cc:482] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:25.954484 10629 tablet_copy_client.cc:523] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:25.954412 10724 raft_consensus.cc:504] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:25.955734 10724 leader_election.cc:288] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:25.956167 4087 catalog_manager.cc:5576] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447) does not have the latest schema for tablet 66a950effa314a7782f0a0804db0615b (table rebalancer_dynamic_table [id=1d747f548b24460ba76f24e385fa5d81]). Expected version 2 got 1
I20240501 20:10:25.956892 7283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ecc1043af4bc1ab31dfa80ea47bb4" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:25.957157 7283 raft_consensus.cc:2454] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:10:25.957566 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ecc1043af4bc1ab31dfa80ea47bb4" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:25.957811 4499 raft_consensus.cc:2454] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 0.
I20240501 20:10:25.958410 7355 leader_election.cc:302] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, b9957d4086bc4a76b18ed9265a5739d5; no voters:
I20240501 20:10:25.958585 10629 tablet_bootstrap.cc:492] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:25.960410 10720 tablet.cc:1722] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:25.961756 10720 tablet.cc:1620] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530: MemRowSet was empty: no flush needed.
I20240501 20:10:25.961861 10721 tablet.cc:1722] T 66a950effa314a7782f0a0804db0615b P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:25.962682 10721 tablet.cc:1620] T 66a950effa314a7782f0a0804db0615b P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:25.963717 10221 raft_consensus.cc:2781] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:25.963864 10221 raft_consensus.cc:482] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:25.963951 10221 raft_consensus.cc:3037] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:25.967379 10221 raft_consensus.cc:504] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:25.970351 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ecc1043af4bc1ab31dfa80ea47bb4" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:10:25.970587 4498 raft_consensus.cc:3037] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:25.978767 4091 catalog_manager.cc:5576] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941) does not have the latest schema for tablet 66a950effa314a7782f0a0804db0615b (table rebalancer_dynamic_table [id=1d747f548b24460ba76f24e385fa5d81]). Expected version 2 got 1
I20240501 20:10:25.980976 10629 tablet_bootstrap.cc:492] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:25.981415 10629 tablet_bootstrap.cc:492] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530: Bootstrap complete.
I20240501 20:10:25.981694 10629 ts_tablet_manager.cc:1367] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.001s
I20240501 20:10:25.982841 4498 raft_consensus.cc:2454] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:25.982839 10629 raft_consensus.cc:348] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:25.983078 10629 raft_consensus.cc:729] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: LEARNER
I20240501 20:10:25.983296 10629 consensus_queue.cc:260] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:25.984141 10731 tablet.cc:1722] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:25.984797 10731 tablet.cc:1620] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:25.987478 7355 leader_election.cc:302] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, b9957d4086bc4a76b18ed9265a5739d5; no voters:
I20240501 20:10:25.988149 10629 ts_tablet_manager.cc:1398] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:10:25.988166 10728 raft_consensus.cc:2781] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:25.988360 10728 raft_consensus.cc:686] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Becoming Leader. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Running, Role: LEADER
I20240501 20:10:25.988736 10728 consensus_queue.cc:237] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:25.986492 4365 raft_consensus.cc:2932] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:25.994757 7042 tablet_copy_service.cc:342] P ef0a4ada548446b3a531cb39a3c4e38d: Request end of tablet copy session dca57823003e4729b769ca12367d7530-743d7fd2ae014352a5681c2a8ee4447d received from {username='slave'} at 127.29.219.65:40207
I20240501 20:10:25.994920 7042 tablet_copy_service.cc:433] P ef0a4ada548446b3a531cb39a3c4e38d: ending tablet copy session dca57823003e4729b769ca12367d7530-743d7fd2ae014352a5681c2a8ee4447d on tablet 743d7fd2ae014352a5681c2a8ee4447d with peer dca57823003e4729b769ca12367d7530
I20240501 20:10:26.000214 7281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e0ecc1043af4bc1ab31dfa80ea47bb4" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:26.000622 7281 raft_consensus.cc:3037] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:26.003409 7281 raft_consensus.cc:2454] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:26.020455 4090 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=1d747f548b24460ba76f24e385fa5d81] alter complete (version 2)
I20240501 20:10:26.022977 4087 catalog_manager.cc:5576] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237) does not have the latest schema for tablet 66a950effa314a7782f0a0804db0615b (table rebalancer_dynamic_table [id=1d747f548b24460ba76f24e385fa5d81]). Expected version 2 got 1
I20240501 20:10:25.988744 10221 leader_election.cc:288] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:26.038152 10696 raft_consensus.cc:482] T b67250c5773f4c9d9254d0306158ccde P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:26.038350 10696 raft_consensus.cc:504] T b67250c5773f4c9d9254d0306158ccde P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:26.039121 10696 leader_election.cc:288] T b67250c5773f4c9d9254d0306158ccde P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:26.039963 7281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b67250c5773f4c9d9254d0306158ccde" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:26.040589 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b67250c5773f4c9d9254d0306158ccde" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
W20240501 20:10:26.040582 7089 leader_election.cc:341] T b67250c5773f4c9d9254d0306158ccde P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:10:26.041144 7090 leader_election.cc:341] T b67250c5773f4c9d9254d0306158ccde P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:26.041337 7090 leader_election.cc:302] T b67250c5773f4c9d9254d0306158ccde P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db0742ee3860468590eefef6ae6074fd; no voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:26.041666 10696 raft_consensus.cc:2726] T b67250c5773f4c9d9254d0306158ccde P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 20:10:26.055914 7356 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.68:45205, user_credentials={real_user=slave}} blocked reactor thread for 56192.8us
I20240501 20:10:26.056185 10733 tablet_bootstrap.cc:654] T aecca0e22b084a8b95c98ef7e7b90b29 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
W20240501 20:10:26.057363 7355 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.65:42941, user_credentials={real_user=slave}} blocked reactor thread for 58793.1us
I20240501 20:10:26.059855 10725 tablet_bootstrap.cc:492] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:26.060077 10725 ts_tablet_manager.cc:1367] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.257s user 0.083s sys 0.106s
I20240501 20:10:26.060925 10725 raft_consensus.cc:348] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:26.061199 10725 raft_consensus.cc:374] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:26.061312 10725 raft_consensus.cc:729] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:26.061585 10725 consensus_queue.cc:260] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:26.062204 10725 ts_tablet_manager.cc:1398] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:10:26.062536 10725 tablet_bootstrap.cc:492] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:26.065348 10725 tablet_bootstrap.cc:654] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:26.067188 10733 tablet_bootstrap.cc:492] T aecca0e22b084a8b95c98ef7e7b90b29 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:26.067406 10733 ts_tablet_manager.cc:1367] T aecca0e22b084a8b95c98ef7e7b90b29 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.278s user 0.005s sys 0.000s
I20240501 20:10:26.068539 10733 raft_consensus.cc:348] T aecca0e22b084a8b95c98ef7e7b90b29 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:26.068923 10733 raft_consensus.cc:374] T aecca0e22b084a8b95c98ef7e7b90b29 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:26.069079 10733 raft_consensus.cc:729] T aecca0e22b084a8b95c98ef7e7b90b29 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:26.069402 10733 consensus_queue.cc:260] T aecca0e22b084a8b95c98ef7e7b90b29 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:26.070356 10733 ts_tablet_manager.cc:1398] T aecca0e22b084a8b95c98ef7e7b90b29 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:26.071138 10733 tablet_bootstrap.cc:492] T b67250c5773f4c9d9254d0306158ccde P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:26.086442 4090 catalog_manager.cc:5484] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72). New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:26.109879 10733 tablet_bootstrap.cc:654] T b67250c5773f4c9d9254d0306158ccde P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:26.118522 10725 tablet_bootstrap.cc:492] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:26.118777 10725 ts_tablet_manager.cc:1367] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.056s user 0.015s sys 0.031s
I20240501 20:10:26.119697 10725 raft_consensus.cc:348] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:26.120005 10725 raft_consensus.cc:374] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:26.120141 10725 raft_consensus.cc:729] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:26.120399 10725 consensus_queue.cc:260] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:26.127141 10725 ts_tablet_manager.cc:1398] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240501 20:10:26.141528 10733 tablet_bootstrap.cc:492] T b67250c5773f4c9d9254d0306158ccde P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:26.141813 10733 ts_tablet_manager.cc:1367] T b67250c5773f4c9d9254d0306158ccde P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.071s user 0.006s sys 0.005s
I20240501 20:10:26.142786 10733 raft_consensus.cc:348] T b67250c5773f4c9d9254d0306158ccde P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:26.143050 10733 raft_consensus.cc:374] T b67250c5773f4c9d9254d0306158ccde P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:26.143123 10733 raft_consensus.cc:729] T b67250c5773f4c9d9254d0306158ccde P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:26.143342 10733 consensus_queue.cc:260] T b67250c5773f4c9d9254d0306158ccde P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:26.149613 10733 ts_tablet_manager.cc:1398] T b67250c5773f4c9d9254d0306158ccde P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 20:10:26.157917 4084 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:26.167361 10721 tablet.cc:1722] T 66a950effa314a7782f0a0804db0615b P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:26.167496 10720 tablet.cc:1722] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:26.178362 10731 tablet.cc:1722] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:26.179345 4084 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table_new_name [id=1d747f548b24460ba76f24e385fa5d81] alter complete (version 3)
I20240501 20:10:26.221866 4234 raft_consensus.cc:1203] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:26.291787 4084 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:26.291994 4084 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:26.295718 4084 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 66a950effa314a7782f0a0804db0615b
I20240501 20:10:26.296816 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet 66a950effa314a7782f0a0804db0615b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:26 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:26.296967 7001 tablet_service.cc:1514] Processing DeleteTablet for tablet 66a950effa314a7782f0a0804db0615b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:26 UTC) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:26.297132 7267 tablet_service.cc:1514] Processing DeleteTablet for tablet 66a950effa314a7782f0a0804db0615b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:26 UTC) from {username='slave'} at 127.0.0.1:54166
I20240501 20:10:26.311653 10797 tablet_replica.cc:330] T 66a950effa314a7782f0a0804db0615b P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:26.311949 10797 raft_consensus.cc:2229] T 66a950effa314a7782f0a0804db0615b P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:26.312304 10797 raft_consensus.cc:2258] T 66a950effa314a7782f0a0804db0615b P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:26.312343 10796 tablet_replica.cc:330] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d: stopping tablet replica
I20240501 20:10:26.313642 10797 ts_tablet_manager.cc:1875] T 66a950effa314a7782f0a0804db0615b P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:26.313956 10796 raft_consensus.cc:2229] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:26.314791 10796 raft_consensus.cc:2258] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:26.316336 10796 ts_tablet_manager.cc:1875] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:26.318950 10795 tablet_replica.cc:330] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:26.319164 10795 raft_consensus.cc:2229] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:26.319941 10795 raft_consensus.cc:2258] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:26.321714 10795 ts_tablet_manager.cc:1875] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:26.322583 10797 ts_tablet_manager.cc:1888] T 66a950effa314a7782f0a0804db0615b P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:26.322800 10797 log.cc:1196] T 66a950effa314a7782f0a0804db0615b P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/66a950effa314a7782f0a0804db0615b
I20240501 20:10:26.323382 10797 ts_tablet_manager.cc:1909] T 66a950effa314a7782f0a0804db0615b P bceeae1c79964acabdae8ff771bf0c97: Deleting consensus metadata
I20240501 20:10:26.324550 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet 66a950effa314a7782f0a0804db0615b (table rebalancer_dynamic_table_new_name [id=1d747f548b24460ba76f24e385fa5d81]) successfully deleted
I20240501 20:10:26.325152 4234 consensus_queue.cc:237] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } }
I20240501 20:10:26.328040 4363 raft_consensus.cc:1261] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:26.329102 7281 raft_consensus.cc:1261] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:26.331363 10714 consensus_queue.cc:1035] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:26.331779 10714 consensus_queue.cc:1035] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:26.333967 10795 ts_tablet_manager.cc:1888] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:26.334113 10795 log.cc:1196] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/66a950effa314a7782f0a0804db0615b
I20240501 20:10:26.334715 10795 ts_tablet_manager.cc:1909] T 66a950effa314a7782f0a0804db0615b P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:26.337945 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet 66a950effa314a7782f0a0804db0615b (table rebalancer_dynamic_table_new_name [id=1d747f548b24460ba76f24e385fa5d81]) successfully deleted
I20240501 20:10:26.339608 10796 ts_tablet_manager.cc:1888] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:26.339761 10796 log.cc:1196] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/wals/66a950effa314a7782f0a0804db0615b
I20240501 20:10:26.340237 10796 ts_tablet_manager.cc:1909] T 66a950effa314a7782f0a0804db0615b P ef0a4ada548446b3a531cb39a3c4e38d: Deleting consensus metadata
I20240501 20:10:26.342808 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet 66a950effa314a7782f0a0804db0615b (table rebalancer_dynamic_table_new_name [id=1d747f548b24460ba76f24e385fa5d81]) successfully deleted
I20240501 20:10:26.351090 10673 raft_consensus.cc:2932] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:26.353040 4363 raft_consensus.cc:2932] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:26.356312 7283 raft_consensus.cc:2932] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:26.359627 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6cb1ca3dbf614f94b39e8ddd28ab04fc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:26.364714 4084 catalog_manager.cc:5484] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:26.372833 7134 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cb1ca3dbf614f94b39e8ddd28ab04fc with delete_type TABLET_DATA_TOMBSTONED (TS db0742ee3860468590eefef6ae6074fd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52036
I20240501 20:10:26.374315 10761 raft_consensus.cc:482] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:26.374552 10800 tablet_replica.cc:330] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P db0742ee3860468590eefef6ae6074fd: stopping tablet replica
I20240501 20:10:26.374485 10761 raft_consensus.cc:504] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:26.374867 10800 raft_consensus.cc:2229] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:26.375135 10800 raft_consensus.cc:2258] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:26.376230 10800 ts_tablet_manager.cc:1875] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P db0742ee3860468590eefef6ae6074fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:26.378895 10761 leader_election.cc:288] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:26.378803 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecca0e22b084a8b95c98ef7e7b90b29" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:26.379078 4498 raft_consensus.cc:2454] T aecca0e22b084a8b95c98ef7e7b90b29 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:26.379640 7222 leader_election.cc:302] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:26.380900 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecca0e22b084a8b95c98ef7e7b90b29" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:26.381242 7020 raft_consensus.cc:2454] T aecca0e22b084a8b95c98ef7e7b90b29 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:26.382503 10712 raft_consensus.cc:2781] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:26.382602 10712 raft_consensus.cc:482] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:26.382736 10712 raft_consensus.cc:3037] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:26.384399 10800 ts_tablet_manager.cc:1888] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P db0742ee3860468590eefef6ae6074fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:26.384605 10800 log.cc:1196] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P db0742ee3860468590eefef6ae6074fd: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/wals/6cb1ca3dbf614f94b39e8ddd28ab04fc
I20240501 20:10:26.385546 10712 raft_consensus.cc:504] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:26.386343 10712 leader_election.cc:288] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:26.386771 4076 catalog_manager.cc:4830] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): tablet 6cb1ca3dbf614f94b39e8ddd28ab04fc (table rebalancer_extra_table_36 [id=2cb1d6b4f2204c5c93a6b7b02e2c8a05]) successfully deleted
I20240501 20:10:26.391372 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecca0e22b084a8b95c98ef7e7b90b29" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:10:26.391587 4498 raft_consensus.cc:3037] T aecca0e22b084a8b95c98ef7e7b90b29 P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:26.394194 4498 raft_consensus.cc:2454] T aecca0e22b084a8b95c98ef7e7b90b29 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:26.396211 10712 raft_consensus.cc:482] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:26.396279 7222 leader_election.cc:302] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:26.396436 10712 raft_consensus.cc:504] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:26.397254 10712 leader_election.cc:288] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:26.397442 10712 raft_consensus.cc:2781] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:26.397573 10712 raft_consensus.cc:686] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Becoming Leader. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Running, Role: LEADER
I20240501 20:10:26.397874 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aecca0e22b084a8b95c98ef7e7b90b29" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:26.397823 10712 consensus_queue.cc:237] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:26.398102 7020 raft_consensus.cc:3037] T aecca0e22b084a8b95c98ef7e7b90b29 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:26.398216 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b67250c5773f4c9d9254d0306158ccde" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:26.398366 7019 raft_consensus.cc:2454] T b67250c5773f4c9d9254d0306158ccde P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:26.399943 4087 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:26.401258 7020 raft_consensus.cc:2454] T aecca0e22b084a8b95c98ef7e7b90b29 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:26.402236 7222 leader_election.cc:302] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:26.402577 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b67250c5773f4c9d9254d0306158ccde" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:26.402827 7153 raft_consensus.cc:2454] T b67250c5773f4c9d9254d0306158ccde P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 0.
I20240501 20:10:26.405462 10712 raft_consensus.cc:2781] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:26.405572 10712 raft_consensus.cc:482] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:26.405650 10712 raft_consensus.cc:3037] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:26.407034 4090 catalog_manager.cc:5484] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71). New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:26.409488 10712 raft_consensus.cc:504] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:26.410394 10712 leader_election.cc:288] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:26.411172 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b67250c5773f4c9d9254d0306158ccde" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:26.411487 7020 raft_consensus.cc:3037] T b67250c5773f4c9d9254d0306158ccde P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:26.412813 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b67250c5773f4c9d9254d0306158ccde" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:26.413010 7153 raft_consensus.cc:3037] T b67250c5773f4c9d9254d0306158ccde P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:26.415357 4346 tablet_service.cc:1467] Processing CreateTablet for tablet 87c0a0f5a5094b818de6d87935cce7ce (DEFAULT_TABLE table=rebalancer_dynamic_table [id=be88a358b513433fa5e39f2e931bd198]), partition=
I20240501 20:10:26.415511 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 87c0a0f5a5094b818de6d87935cce7ce (DEFAULT_TABLE table=rebalancer_dynamic_table [id=be88a358b513433fa5e39f2e931bd198]), partition=
I20240501 20:10:26.415732 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c0a0f5a5094b818de6d87935cce7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:26.415845 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c0a0f5a5094b818de6d87935cce7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:26.416193 7020 raft_consensus.cc:2454] T b67250c5773f4c9d9254d0306158ccde P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:26.416813 7222 leader_election.cc:302] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:26.419783 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 87c0a0f5a5094b818de6d87935cce7ce (DEFAULT_TABLE table=rebalancer_dynamic_table [id=be88a358b513433fa5e39f2e931bd198]), partition=
I20240501 20:10:26.420188 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87c0a0f5a5094b818de6d87935cce7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:26.422436 7153 raft_consensus.cc:2454] T b67250c5773f4c9d9254d0306158ccde P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:26.425097 10712 raft_consensus.cc:2781] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:26.425655 10712 raft_consensus.cc:686] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Becoming Leader. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Running, Role: LEADER
I20240501 20:10:26.426033 10712 consensus_queue.cc:237] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:26.437637 10810 tablet_bootstrap.cc:492] T 87c0a0f5a5094b818de6d87935cce7ce P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:26.440784 4090 catalog_manager.cc:5484] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71). New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:26.443923 10807 tablet_bootstrap.cc:492] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:26.444133 10810 tablet_bootstrap.cc:654] T 87c0a0f5a5094b818de6d87935cce7ce P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:26.446576 10807 tablet_bootstrap.cc:654] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:26.446758 10809 tablet_bootstrap.cc:492] T 87c0a0f5a5094b818de6d87935cce7ce P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:26.448215 10809 tablet_bootstrap.cc:654] T 87c0a0f5a5094b818de6d87935cce7ce P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:26.450038 10810 tablet_bootstrap.cc:492] T 87c0a0f5a5094b818de6d87935cce7ce P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:26.450186 10810 ts_tablet_manager.cc:1367] T 87c0a0f5a5094b818de6d87935cce7ce P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240501 20:10:26.450392 10809 tablet_bootstrap.cc:492] T 87c0a0f5a5094b818de6d87935cce7ce P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:26.450938 10810 raft_consensus.cc:348] T 87c0a0f5a5094b818de6d87935cce7ce P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:26.450960 10809 ts_tablet_manager.cc:1367] T 87c0a0f5a5094b818de6d87935cce7ce P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 20:10:26.451447 10807 tablet_bootstrap.cc:492] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:10:26.451642 10807 ts_tablet_manager.cc:1367] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20240501 20:10:26.451776 10809 raft_consensus.cc:348] T 87c0a0f5a5094b818de6d87935cce7ce P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:26.452051 10809 raft_consensus.cc:374] T 87c0a0f5a5094b818de6d87935cce7ce P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:26.452147 10809 raft_consensus.cc:729] T 87c0a0f5a5094b818de6d87935cce7ce P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:26.452445 10809 consensus_queue.cc:260] T 87c0a0f5a5094b818de6d87935cce7ce P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:26.452430 10807 raft_consensus.cc:348] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:26.452656 10807 raft_consensus.cc:374] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:26.452764 10810 raft_consensus.cc:374] T 87c0a0f5a5094b818de6d87935cce7ce P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:26.452881 10810 raft_consensus.cc:729] T 87c0a0f5a5094b818de6d87935cce7ce P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:26.453045 10807 raft_consensus.cc:729] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:10:26.453410 10810 consensus_queue.cc:260] T 87c0a0f5a5094b818de6d87935cce7ce P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:26.453776 10807 consensus_queue.cc:260] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:26.454375 10810 ts_tablet_manager.cc:1398] T 87c0a0f5a5094b818de6d87935cce7ce P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:10:26.454960 10807 ts_tablet_manager.cc:1398] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:10:26.460258 10809 ts_tablet_manager.cc:1398] T 87c0a0f5a5094b818de6d87935cce7ce P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240501 20:10:26.485203 10728 consensus_queue.cc:1035] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:26.504973 4234 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dfaa3794205d482c90aa4f860fa99cf3"
dest_uuid: "dca57823003e4729b769ca12367d7530"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38616
I20240501 20:10:26.505182 4234 raft_consensus.cc:595] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Received request to transfer leadership
I20240501 20:10:26.505497 10728 consensus_queue.cc:1035] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:26.523175 10496 raft_consensus.cc:482] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:26.523339 10496 raft_consensus.cc:504] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:26.524031 10496 leader_election.cc:288] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:26.524482 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c0a0f5a5094b818de6d87935cce7ce" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:26.524673 4627 raft_consensus.cc:2454] T 87c0a0f5a5094b818de6d87935cce7ce P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:10:26.525038 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c0a0f5a5094b818de6d87935cce7ce" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:26.525250 4234 raft_consensus.cc:2454] T 87c0a0f5a5094b818de6d87935cce7ce P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 0.
I20240501 20:10:26.525527 4302 leader_election.cc:302] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 1d34e18a1d394d3f91c9f50d1c7bea4e; no voters:
I20240501 20:10:26.525995 10496 raft_consensus.cc:2781] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:26.526109 10496 raft_consensus.cc:482] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:26.526198 10496 raft_consensus.cc:3037] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:26.529834 10496 raft_consensus.cc:504] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:26.530701 10496 leader_election.cc:288] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:26.531071 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c0a0f5a5094b818de6d87935cce7ce" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:10:26.531129 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87c0a0f5a5094b818de6d87935cce7ce" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:10:26.531286 4627 raft_consensus.cc:3037] T 87c0a0f5a5094b818de6d87935cce7ce P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:26.531286 4234 raft_consensus.cc:3037] T 87c0a0f5a5094b818de6d87935cce7ce P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:26.534027 4234 raft_consensus.cc:2454] T 87c0a0f5a5094b818de6d87935cce7ce P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:10:26.534032 4627 raft_consensus.cc:2454] T 87c0a0f5a5094b818de6d87935cce7ce P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:10:26.534554 4301 leader_election.cc:302] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:26.535182 10496 raft_consensus.cc:2781] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:26.535734 10496 raft_consensus.cc:686] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Becoming Leader. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Running, Role: LEADER
I20240501 20:10:26.536033 10496 consensus_queue.cc:237] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:26.537045 4234 consensus_queue.cc:237] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:26.539543 7417 raft_consensus.cc:1261] T d13178f10db0435891b97cebaed6dd5d P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:26.539656 7283 raft_consensus.cc:1261] T d13178f10db0435891b97cebaed6dd5d P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:26.540210 10673 consensus_queue.cc:1035] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:26.540470 4169 consensus_peers.cc:485] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d13178f10db0435891b97cebaed6dd5d. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:26.541627 10673 consensus_queue.cc:1035] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:26.542590 4090 catalog_manager.cc:5484] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a reported cstate change: term changed from 0 to 1, leader changed from <none> to 10b47782558c41ee91f5e164a372049a (127.29.219.66). New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:26.544754 10673 raft_consensus.cc:2932] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:26.548453 7417 raft_consensus.cc:2932] T d13178f10db0435891b97cebaed6dd5d P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:26.548705 7283 raft_consensus.cc:2932] T d13178f10db0435891b97cebaed6dd5d P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:26.553503 4090 catalog_manager.cc:5484] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:26.572630 7417 consensus_queue.cc:237] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:26.574900 4627 raft_consensus.cc:1261] T c5b4fbeda51f40b8ac54baf5fb241710 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:26.575423 10728 consensus_queue.cc:1035] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:26.575604 4496 raft_consensus.cc:1261] T c5b4fbeda51f40b8ac54baf5fb241710 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:26.575747 7355 consensus_peers.cc:485] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5b4fbeda51f40b8ac54baf5fb241710. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:26.576202 10728 consensus_queue.cc:1035] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:26.578349 10724 raft_consensus.cc:2932] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:26.578979 4627 raft_consensus.cc:2932] T c5b4fbeda51f40b8ac54baf5fb241710 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:26.581569 4496 raft_consensus.cc:2932] T c5b4fbeda51f40b8ac54baf5fb241710 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:26.586285 4087 catalog_manager.cc:5484] T c5b4fbeda51f40b8ac54baf5fb241710 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:26.623706 10350 raft_consensus.cc:979] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530: : Instructing follower bceeae1c79964acabdae8ff771bf0c97 to start an election
I20240501 20:10:26.623860 10350 raft_consensus.cc:1067] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Signalling peer bceeae1c79964acabdae8ff771bf0c97 to start an election
I20240501 20:10:26.624696 7283 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dfaa3794205d482c90aa4f860fa99cf3"
dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
from {username='slave'} at 127.29.219.65:33781
I20240501 20:10:26.625000 7283 raft_consensus.cc:482] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:10:26.625133 7283 raft_consensus.cc:3037] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:26.629421 7283 raft_consensus.cc:504] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:26.630795 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfaa3794205d482c90aa4f860fa99cf3" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:10:26.630988 4234 raft_consensus.cc:3032] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:10:26.631067 4234 raft_consensus.cc:729] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:10:26.631300 4234 consensus_queue.cc:260] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:26.631630 4234 raft_consensus.cc:3037] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:26.633745 4234 raft_consensus.cc:2454] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 2.
I20240501 20:10:26.634641 7283 leader_election.cc:288] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 election: Requested vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:26.635339 4496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfaa3794205d482c90aa4f860fa99cf3" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:10:26.635546 4496 raft_consensus.cc:3037] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:26.638028 4496 raft_consensus.cc:2454] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 2.
I20240501 20:10:26.641021 7222 leader_election.cc:302] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, bceeae1c79964acabdae8ff771bf0c97, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:26.641021 7020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfaa3794205d482c90aa4f860fa99cf3" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:26.641244 7020 raft_consensus.cc:3037] T dfaa3794205d482c90aa4f860fa99cf3 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:26.641407 10712 raft_consensus.cc:2781] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:26.641554 10712 raft_consensus.cc:686] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [term 2 LEADER]: Becoming Leader. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Running, Role: LEADER
I20240501 20:10:26.641849 10712 consensus_queue.cc:237] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:26.644447 7020 raft_consensus.cc:2454] T dfaa3794205d482c90aa4f860fa99cf3 P ef0a4ada548446b3a531cb39a3c4e38d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 2.
I20240501 20:10:26.648530 7153 consensus_queue.cc:237] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:26.660384 7281 raft_consensus.cc:1261] T 797b2374d4214c8eb39982c339a7fbad P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:26.660370 4090 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:26.660938 10648 consensus_queue.cc:1035] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:26.661801 7090 consensus_peers.cc:485] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 797b2374d4214c8eb39982c339a7fbad. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:26.664417 10648 raft_consensus.cc:2932] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:26.665319 7417 raft_consensus.cc:1261] T 797b2374d4214c8eb39982c339a7fbad P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:26.666414 4084 catalog_manager.cc:5484] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: term changed from 1 to 2, leader changed from dca57823003e4729b769ca12367d7530 (127.29.219.65) to bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71). New cstate: current_term: 2 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:26.670305 10696 consensus_queue.cc:1035] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:26.671314 7281 raft_consensus.cc:2932] T 797b2374d4214c8eb39982c339a7fbad P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:26.679711 10701 raft_consensus.cc:1050] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d: attempting to promote NON_VOTER dca57823003e4729b769ca12367d7530 to VOTER
I20240501 20:10:26.680585 10701 consensus_queue.cc:237] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:26.685700 4084 catalog_manager.cc:5484] T 797b2374d4214c8eb39982c339a7fbad P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index -1 to 2, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:26.697293 4627 raft_consensus.cc:1261] T 87c0a0f5a5094b818de6d87935cce7ce P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:26.698303 10717 consensus_queue.cc:1035] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:26.711866 4090 catalog_manager.cc:5576] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205) does not have the latest schema for tablet 87c0a0f5a5094b818de6d87935cce7ce (table rebalancer_dynamic_table [id=be88a358b513433fa5e39f2e931bd198]). Expected version 1 got 0
I20240501 20:10:26.718372 10869 mvcc.cc:205] Tried to move back new op lower bound from 7022977952539394048 to 7022977951894511616. Current Snapshot: MvccSnapshot[applied={T|T < 7022977952539394048}]
I20240501 20:10:26.721769 7420 raft_consensus.cc:1261] T 743d7fd2ae014352a5681c2a8ee4447d P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:26.721949 4628 raft_consensus.cc:1261] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:26.722635 10368 consensus_queue.cc:1035] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:26.723136 4233 raft_consensus.cc:1261] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:26.723176 10368 consensus_queue.cc:1035] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:26.727563 10368 consensus_queue.cc:1035] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:26.729576 10876 tablet.cc:1722] T 87c0a0f5a5094b818de6d87935cce7ce P 1d34e18a1d394d3f91c9f50d1c7bea4e: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:26.730193 10876 tablet.cc:1620] T 87c0a0f5a5094b818de6d87935cce7ce P 1d34e18a1d394d3f91c9f50d1c7bea4e: MemRowSet was empty: no flush needed.
I20240501 20:10:26.740576 4090 catalog_manager.cc:5576] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205) does not have the latest schema for tablet 87c0a0f5a5094b818de6d87935cce7ce (table rebalancer_dynamic_table [id=be88a358b513433fa5e39f2e931bd198]). Expected version 1 got 0
I20240501 20:10:26.748433 4234 raft_consensus.cc:1261] T 87c0a0f5a5094b818de6d87935cce7ce P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:26.749293 10496 consensus_queue.cc:1035] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:26.761098 10884 tablet.cc:1722] T 87c0a0f5a5094b818de6d87935cce7ce P dca57823003e4729b769ca12367d7530: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:26.761910 10884 tablet.cc:1620] T 87c0a0f5a5094b818de6d87935cce7ce P dca57823003e4729b769ca12367d7530: MemRowSet was empty: no flush needed.
I20240501 20:10:26.764875 4090 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=be88a358b513433fa5e39f2e931bd198] alter complete (version 1)
I20240501 20:10:26.766505 4084 catalog_manager.cc:5576] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941) does not have the latest schema for tablet 87c0a0f5a5094b818de6d87935cce7ce (table rebalancer_dynamic_table [id=be88a358b513433fa5e39f2e931bd198]). Expected version 1 got 0
I20240501 20:10:26.795332 10881 tablet.cc:1722] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:26.795990 10881 tablet.cc:1620] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a: MemRowSet was empty: no flush needed.
I20240501 20:10:26.825974 7420 raft_consensus.cc:2932] T 743d7fd2ae014352a5681c2a8ee4447d P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:26.826385 4233 raft_consensus.cc:2932] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:26.831943 7417 raft_consensus.cc:2932] T 797b2374d4214c8eb39982c339a7fbad P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:26.833287 4084 catalog_manager.cc:5484] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:26.835258 10368 raft_consensus.cc:2932] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:26.846211 4628 raft_consensus.cc:2932] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:26.848706 7022 consensus_queue.cc:237] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:26.858947 4233 raft_consensus.cc:1261] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:26.859663 4628 raft_consensus.cc:1261] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:26.859602 10877 consensus_queue.cc:1035] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 20:10:26.861342 10368 consensus_queue.cc:1035] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:26.864225 4363 consensus_queue.cc:237] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:26.864038 10700 raft_consensus.cc:2932] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:26.867430 4628 raft_consensus.cc:2932] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:26.867786 7153 raft_consensus.cc:1261] T 26ed0760e9074bd896e55e99756602bb P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:26.867856 7283 raft_consensus.cc:1261] T 26ed0760e9074bd896e55e99756602bb P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:26.868338 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 743d7fd2ae014352a5681c2a8ee4447d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:26.868515 10496 consensus_queue.cc:1035] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:26.869175 10885 consensus_queue.cc:1035] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:26.869448 4301 consensus_peers.cc:485] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26ed0760e9074bd896e55e99756602bb. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:26.873678 4084 catalog_manager.cc:5484] T 743d7fd2ae014352a5681c2a8ee4447d P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:26.881306 4090 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:26.884459 7400 tablet_service.cc:1514] Processing DeleteTablet for tablet 743d7fd2ae014352a5681c2a8ee4447d with delete_type TABLET_DATA_TOMBSTONED (TS b9957d4086bc4a76b18ed9265a5739d5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40182
I20240501 20:10:26.889398 10885 raft_consensus.cc:2932] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:26.892947 10881 tablet.cc:1722] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:26.893448 10881 tablet.cc:1620] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a: MemRowSet was empty: no flush needed.
I20240501 20:10:26.894920 10876 tablet.cc:1722] T 87c0a0f5a5094b818de6d87935cce7ce P 1d34e18a1d394d3f91c9f50d1c7bea4e: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:26.895359 10876 tablet.cc:1620] T 87c0a0f5a5094b818de6d87935cce7ce P 1d34e18a1d394d3f91c9f50d1c7bea4e: MemRowSet was empty: no flush needed.
I20240501 20:10:26.896162 4234 raft_consensus.cc:2932] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:26.896606 7152 raft_consensus.cc:2932] T 26ed0760e9074bd896e55e99756602bb P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:26.897682 10884 tablet.cc:1722] T 87c0a0f5a5094b818de6d87935cce7ce P dca57823003e4729b769ca12367d7530: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:26.898111 10884 tablet.cc:1620] T 87c0a0f5a5094b818de6d87935cce7ce P dca57823003e4729b769ca12367d7530: MemRowSet was empty: no flush needed.
I20240501 20:10:26.909217 4087 catalog_manager.cc:5484] T 26ed0760e9074bd896e55e99756602bb P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:26.911264 4091 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=be88a358b513433fa5e39f2e931bd198] alter complete (version 2)
I20240501 20:10:26.916440 7283 raft_consensus.cc:2932] T 26ed0760e9074bd896e55e99756602bb P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:26.927011 10712 consensus_queue.cc:1035] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:26.930771 10896 tablet_replica.cc:330] T 743d7fd2ae014352a5681c2a8ee4447d P b9957d4086bc4a76b18ed9265a5739d5: stopping tablet replica
I20240501 20:10:26.933184 10896 raft_consensus.cc:2229] T 743d7fd2ae014352a5681c2a8ee4447d P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:26.933710 10896 raft_consensus.cc:2258] T 743d7fd2ae014352a5681c2a8ee4447d P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:26.934784 10896 ts_tablet_manager.cc:1875] T 743d7fd2ae014352a5681c2a8ee4447d P b9957d4086bc4a76b18ed9265a5739d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:26.935334 10761 consensus_queue.cc:1035] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:26.947988 10896 ts_tablet_manager.cc:1888] T 743d7fd2ae014352a5681c2a8ee4447d P b9957d4086bc4a76b18ed9265a5739d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:26.948215 10896 log.cc:1196] T 743d7fd2ae014352a5681c2a8ee4447d P b9957d4086bc4a76b18ed9265a5739d5: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/wals/743d7fd2ae014352a5681c2a8ee4447d
I20240501 20:10:26.952251 4079 catalog_manager.cc:4830] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): tablet 743d7fd2ae014352a5681c2a8ee4447d (table rebalancer_extra_table_32 [id=338f819ddccb42a3ae45044c701a6eb5]) successfully deleted
I20240501 20:10:26.960661 7022 consensus_queue.cc:237] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
W20240501 20:10:26.963562 6957 consensus_peers.cc:485] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48528ec242974de4ac20eb0161f0cb05. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:26.963990 7417 raft_consensus.cc:1261] T 48528ec242974de4ac20eb0161f0cb05 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:26.964521 7153 raft_consensus.cc:1261] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:26.964964 10701 consensus_queue.cc:1035] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:26.965272 10877 consensus_queue.cc:1035] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:26.968609 10893 raft_consensus.cc:2932] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:26.969223 7417 raft_consensus.cc:2932] T 48528ec242974de4ac20eb0161f0cb05 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:26.975674 4090 catalog_manager.cc:5484] T 48528ec242974de4ac20eb0161f0cb05 P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:26.977193 7153 raft_consensus.cc:2932] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:26.979041 10761 consensus_queue.cc:1035] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:26.983124 4084 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:26.999779 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3bb8370e7c41dc9cf7f1c835ac275b (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=19d9a775c8ed49b4919760e18d418208]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:27.000247 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3bb8370e7c41dc9cf7f1c835ac275b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.000689 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3bb8370e7c41dc9cf7f1c835ac275b (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=19d9a775c8ed49b4919760e18d418208]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:27.001063 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3bb8370e7c41dc9cf7f1c835ac275b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.001794 4346 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3bb8370e7c41dc9cf7f1c835ac275b (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=19d9a775c8ed49b4919760e18d418208]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:27.002193 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3bb8370e7c41dc9cf7f1c835ac275b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.003101 4609 tablet_service.cc:1467] Processing CreateTablet for tablet 6ced3d988d474a8e8ee8f0a1fb4cc88e (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=19d9a775c8ed49b4919760e18d418208]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:27.003176 4611 tablet_service.cc:1467] Processing CreateTablet for tablet da878c45e7184273ba6d3497e9eef35d (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=19d9a775c8ed49b4919760e18d418208]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:27.003196 7267 tablet_service.cc:1467] Processing CreateTablet for tablet 6ced3d988d474a8e8ee8f0a1fb4cc88e (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=19d9a775c8ed49b4919760e18d418208]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:27.003443 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ced3d988d474a8e8ee8f0a1fb4cc88e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.003556 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ced3d988d474a8e8ee8f0a1fb4cc88e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.007457 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da878c45e7184273ba6d3497e9eef35d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.008850 4478 tablet_service.cc:1467] Processing CreateTablet for tablet 6ced3d988d474a8e8ee8f0a1fb4cc88e (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=19d9a775c8ed49b4919760e18d418208]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:27.009244 4478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ced3d988d474a8e8ee8f0a1fb4cc88e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.011286 10712 consensus_queue.cc:1035] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:27.012671 7001 tablet_service.cc:1467] Processing CreateTablet for tablet da878c45e7184273ba6d3497e9eef35d (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=19d9a775c8ed49b4919760e18d418208]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:27.013151 7001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da878c45e7184273ba6d3497e9eef35d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.015653 4089 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:27.028789 10881 tablet.cc:1722] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:27.029685 7134 tablet_service.cc:1467] Processing CreateTablet for tablet da878c45e7184273ba6d3497e9eef35d (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=19d9a775c8ed49b4919760e18d418208]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:27.030203 7134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da878c45e7184273ba6d3497e9eef35d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.038497 10884 tablet.cc:1722] T 87c0a0f5a5094b818de6d87935cce7ce P dca57823003e4729b769ca12367d7530: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:27.041185 10876 tablet.cc:1722] T 87c0a0f5a5094b818de6d87935cce7ce P 1d34e18a1d394d3f91c9f50d1c7bea4e: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:27.046013 10918 tablet_bootstrap.cc:492] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:27.053452 10917 tablet_bootstrap.cc:492] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:27.053748 4234 raft_consensus.cc:1261] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [term 2 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:10:27.054857 10712 consensus_queue.cc:1035] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:27.056738 10921 tablet_bootstrap.cc:492] T da878c45e7184273ba6d3497e9eef35d P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:27.057433 10917 tablet_bootstrap.cc:654] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:27.062692 4090 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table_new_name [id=be88a358b513433fa5e39f2e931bd198] alter complete (version 3)
I20240501 20:10:27.066219 10921 tablet_bootstrap.cc:654] T da878c45e7184273ba6d3497e9eef35d P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:27.078953 10921 tablet_bootstrap.cc:492] T da878c45e7184273ba6d3497e9eef35d P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:27.079648 10921 ts_tablet_manager.cc:1367] T da878c45e7184273ba6d3497e9eef35d P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.005s
I20240501 20:10:27.082358 10917 tablet_bootstrap.cc:492] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:27.082664 10917 ts_tablet_manager.cc:1367] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.007s
I20240501 20:10:27.083621 10917 raft_consensus.cc:348] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:27.083957 10917 raft_consensus.cc:374] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:27.084105 10917 raft_consensus.cc:729] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:27.084434 10917 consensus_queue.cc:260] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:27.085579 10923 tablet_bootstrap.cc:492] T da878c45e7184273ba6d3497e9eef35d P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:27.088430 10920 tablet_bootstrap.cc:492] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:27.086535 10919 tablet_bootstrap.cc:492] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:27.088878 10922 tablet_bootstrap.cc:492] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:27.091768 10919 tablet_bootstrap.cc:654] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 3e974c03d3c546f9ab6c91e762a5426b: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:27.092060 10918 tablet_bootstrap.cc:654] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:27.092211 10917 ts_tablet_manager.cc:1398] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240501 20:10:27.094892 10377 raft_consensus.cc:482] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:27.095039 10377 raft_consensus.cc:504] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:27.095952 10377 leader_election.cc:288] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:27.098439 10918 tablet_bootstrap.cc:492] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:27.098338 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da878c45e7184273ba6d3497e9eef35d" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:27.098729 10918 ts_tablet_manager.cc:1367] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.003s
I20240501 20:10:27.099745 10917 tablet_bootstrap.cc:492] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:27.099593 10918 raft_consensus.cc:348] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:27.099829 10918 raft_consensus.cc:374] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:27.101924 10917 tablet_bootstrap.cc:654] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:27.081141 10921 raft_consensus.cc:348] T da878c45e7184273ba6d3497e9eef35d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:27.102293 10921 raft_consensus.cc:374] T da878c45e7184273ba6d3497e9eef35d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:27.102591 10921 raft_consensus.cc:729] T da878c45e7184273ba6d3497e9eef35d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:27.103003 10921 consensus_queue.cc:260] T da878c45e7184273ba6d3497e9eef35d P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:27.103405 7022 raft_consensus.cc:2454] T da878c45e7184273ba6d3497e9eef35d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:10:27.103752 10920 tablet_bootstrap.cc:654] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:27.100010 10918 raft_consensus.cc:729] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:27.104746 10918 consensus_queue.cc:260] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:27.106874 10922 tablet_bootstrap.cc:654] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:27.093364 10923 tablet_bootstrap.cc:654] T da878c45e7184273ba6d3497e9eef35d P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:27.097357 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da878c45e7184273ba6d3497e9eef35d" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:27.109933 10921 ts_tablet_manager.cc:1398] T da878c45e7184273ba6d3497e9eef35d P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.030s user 0.000s sys 0.002s
I20240501 20:10:27.113046 10931 ts_tablet_manager.cc:902] T d13178f10db0435891b97cebaed6dd5d P 3e974c03d3c546f9ab6c91e762a5426b: Initiating tablet copy from peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:27.104058 4564 leader_election.cc:302] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
W20240501 20:10:27.114598 4562 leader_election.cc:341] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:27.115317 10377 raft_consensus.cc:2781] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:27.115418 10377 raft_consensus.cc:482] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:27.115515 10377 raft_consensus.cc:3037] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:27.118280 10377 raft_consensus.cc:504] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:27.121023 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da878c45e7184273ba6d3497e9eef35d" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:27.121286 7022 raft_consensus.cc:3037] T da878c45e7184273ba6d3497e9eef35d P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:27.126847 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da878c45e7184273ba6d3497e9eef35d" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
W20240501 20:10:27.127545 4562 leader_election.cc:341] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:27.127795 10377 leader_election.cc:288] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:27.128129 7022 raft_consensus.cc:2454] T da878c45e7184273ba6d3497e9eef35d P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:10:27.128646 4564 leader_election.cc:302] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, ef0a4ada548446b3a531cb39a3c4e38d; no voters: db0742ee3860468590eefef6ae6074fd
I20240501 20:10:27.129704 10377 raft_consensus.cc:2781] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:27.130276 10377 raft_consensus.cc:686] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Becoming Leader. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Running, Role: LEADER
I20240501 20:10:27.130300 10918 ts_tablet_manager.cc:1398] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.031s user 0.001s sys 0.000s
I20240501 20:10:27.130666 10377 consensus_queue.cc:237] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:27.135257 10920 tablet_bootstrap.cc:492] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:27.137502 10920 ts_tablet_manager.cc:1367] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.049s user 0.006s sys 0.000s
I20240501 20:10:27.138353 10923 tablet_bootstrap.cc:492] T da878c45e7184273ba6d3497e9eef35d P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:27.138732 4090 catalog_manager.cc:5484] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68). New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:27.139317 10923 ts_tablet_manager.cc:1367] T da878c45e7184273ba6d3497e9eef35d P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.003s
I20240501 20:10:27.141405 10923 raft_consensus.cc:348] T da878c45e7184273ba6d3497e9eef35d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:27.141949 10920 raft_consensus.cc:348] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:27.142714 10920 raft_consensus.cc:374] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:27.142841 10923 raft_consensus.cc:374] T da878c45e7184273ba6d3497e9eef35d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:27.143004 10920 raft_consensus.cc:729] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:27.143208 10923 raft_consensus.cc:729] T da878c45e7184273ba6d3497e9eef35d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:27.143872 10923 consensus_queue.cc:260] T da878c45e7184273ba6d3497e9eef35d P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:27.144116 10920 consensus_queue.cc:260] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:27.149797 4234 consensus_queue.cc:237] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:27.151932 10919 tablet_bootstrap.cc:492] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 3e974c03d3c546f9ab6c91e762a5426b: No bootstrap required, opened a new log
I20240501 20:10:27.152117 10919 ts_tablet_manager.cc:1367] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.066s user 0.000s sys 0.004s
I20240501 20:10:27.152179 10931 tablet_copy_client.cc:308] T d13178f10db0435891b97cebaed6dd5d P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.65:42941
I20240501 20:10:27.152992 10919 raft_consensus.cc:348] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:27.153219 10919 raft_consensus.cc:374] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:27.153277 10919 raft_consensus.cc:729] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: FOLLOWER
I20240501 20:10:27.153430 10917 tablet_bootstrap.cc:492] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:27.153587 10917 ts_tablet_manager.cc:1367] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.054s user 0.000s sys 0.006s
I20240501 20:10:27.153476 10919 consensus_queue.cc:260] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:27.154074 10919 ts_tablet_manager.cc:1398] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:10:27.154750 4254 tablet_copy_service.cc:140] P dca57823003e4729b769ca12367d7530: Received BeginTabletCopySession request for tablet d13178f10db0435891b97cebaed6dd5d from peer 3e974c03d3c546f9ab6c91e762a5426b ({username='slave'} at 127.29.219.67:51285)
I20240501 20:10:27.154883 4254 tablet_copy_service.cc:161] P dca57823003e4729b769ca12367d7530: Beginning new tablet copy session on tablet d13178f10db0435891b97cebaed6dd5d from peer 3e974c03d3c546f9ab6c91e762a5426b at {username='slave'} at 127.29.219.67:51285: session id = 3e974c03d3c546f9ab6c91e762a5426b-d13178f10db0435891b97cebaed6dd5d
I20240501 20:10:27.155191 10917 raft_consensus.cc:348] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:27.155433 10917 raft_consensus.cc:374] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:27.155506 10917 raft_consensus.cc:729] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:27.155768 10917 consensus_queue.cc:260] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:27.157672 4254 tablet_copy_source_session.cc:205] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:27.159323 10917 ts_tablet_manager.cc:1398] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240501 20:10:27.163084 10922 tablet_bootstrap.cc:492] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:10:27.163231 10922 ts_tablet_manager.cc:1367] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.074s user 0.006s sys 0.000s
I20240501 20:10:27.164039 10922 raft_consensus.cc:348] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:27.164247 10922 raft_consensus.cc:374] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:27.164317 10922 raft_consensus.cc:729] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:10:27.164566 10922 consensus_queue.cc:260] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:27.166071 10917 tablet_bootstrap.cc:492] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:27.166905 10922 ts_tablet_manager.cc:1398] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 20:10:27.167500 10923 ts_tablet_manager.cc:1398] T da878c45e7184273ba6d3497e9eef35d P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.028s user 0.000s sys 0.002s
I20240501 20:10:27.168109 10920 ts_tablet_manager.cc:1398] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20240501 20:10:27.170990 10917 tablet_bootstrap.cc:654] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:27.172806 4091 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:27.173030 4091 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:27.175031 10931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d13178f10db0435891b97cebaed6dd5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.176587 4091 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 87c0a0f5a5094b818de6d87935cce7ce
I20240501 20:10:27.177465 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet 87c0a0f5a5094b818de6d87935cce7ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:27 UTC) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:27.178059 4346 tablet_service.cc:1514] Processing DeleteTablet for tablet 87c0a0f5a5094b818de6d87935cce7ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:27 UTC) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:27.178086 4611 tablet_service.cc:1514] Processing DeleteTablet for tablet 87c0a0f5a5094b818de6d87935cce7ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:27 UTC) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:27.201687 10946 tablet_replica.cc:330] T 87c0a0f5a5094b818de6d87935cce7ce P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:27.202108 10946 raft_consensus.cc:2229] T 87c0a0f5a5094b818de6d87935cce7ce P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:27.202183 10931 tablet_copy_client.cc:783] T d13178f10db0435891b97cebaed6dd5d P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:27.202478 10931 tablet_copy_client.cc:647] T d13178f10db0435891b97cebaed6dd5d P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:27.202649 10946 raft_consensus.cc:2258] T 87c0a0f5a5094b818de6d87935cce7ce P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:27.204882 10945 tablet_replica.cc:330] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:27.204934 10946 ts_tablet_manager.cc:1875] T 87c0a0f5a5094b818de6d87935cce7ce P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:27.205238 10945 raft_consensus.cc:2229] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:27.205644 10944 tablet_replica.cc:330] T 87c0a0f5a5094b818de6d87935cce7ce P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:27.206271 10945 raft_consensus.cc:2258] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:27.206449 10931 tablet_copy_client.cc:523] T d13178f10db0435891b97cebaed6dd5d P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:27.206573 10944 raft_consensus.cc:2229] T 87c0a0f5a5094b818de6d87935cce7ce P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:27.206881 10944 raft_consensus.cc:2258] T 87c0a0f5a5094b818de6d87935cce7ce P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:27.210175 10945 ts_tablet_manager.cc:1875] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:27.212138 10944 ts_tablet_manager.cc:1875] T 87c0a0f5a5094b818de6d87935cce7ce P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:27.213182 10947 ts_tablet_manager.cc:902] T c5b4fbeda51f40b8ac54baf5fb241710 P 10b47782558c41ee91f5e164a372049a: Initiating tablet copy from peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:27.214025 10948 ts_tablet_manager.cc:902] T 797b2374d4214c8eb39982c339a7fbad P ef0a4ada548446b3a531cb39a3c4e38d: Initiating tablet copy from peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:27.224242 10931 tablet_bootstrap.cc:492] T d13178f10db0435891b97cebaed6dd5d P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
W20240501 20:10:27.225172 4170 consensus_peers.cc:485] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 -> Peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Couldn't send request to peer 1d34e18a1d394d3f91c9f50d1c7bea4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00ae88161b514e4aa4b8d134a19de001. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:27.227656 10917 tablet_bootstrap.cc:492] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:27.228273 10917 ts_tablet_manager.cc:1367] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.069s user 0.008s sys 0.002s
I20240501 20:10:27.231408 4496 raft_consensus.cc:1261] T 00ae88161b514e4aa4b8d134a19de001 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:27.232287 10672 consensus_queue.cc:1035] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:27.233901 10944 ts_tablet_manager.cc:1888] T 87c0a0f5a5094b818de6d87935cce7ce P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:27.234009 10944 log.cc:1196] T 87c0a0f5a5094b818de6d87935cce7ce P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/87c0a0f5a5094b818de6d87935cce7ce
I20240501 20:10:27.234403 10944 ts_tablet_manager.cc:1909] T 87c0a0f5a5094b818de6d87935cce7ce P dca57823003e4729b769ca12367d7530: Deleting consensus metadata
I20240501 20:10:27.235544 10945 ts_tablet_manager.cc:1888] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:27.235733 10945 log.cc:1196] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/87c0a0f5a5094b818de6d87935cce7ce
I20240501 20:10:27.235846 7020 raft_consensus.cc:1261] T 00ae88161b514e4aa4b8d134a19de001 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:27.236204 10945 ts_tablet_manager.cc:1909] T 87c0a0f5a5094b818de6d87935cce7ce P 10b47782558c41ee91f5e164a372049a: Deleting consensus metadata
I20240501 20:10:27.236498 10917 raft_consensus.cc:348] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:27.236598 10672 consensus_queue.cc:1035] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:27.236761 10917 raft_consensus.cc:374] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:27.236841 10917 raft_consensus.cc:729] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:27.238565 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet 87c0a0f5a5094b818de6d87935cce7ce (table rebalancer_dynamic_table_new_name [id=be88a358b513433fa5e39f2e931bd198]) successfully deleted
I20240501 20:10:27.239025 10917 consensus_queue.cc:260] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:27.239224 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet 87c0a0f5a5094b818de6d87935cce7ce (table rebalancer_dynamic_table_new_name [id=be88a358b513433fa5e39f2e931bd198]) successfully deleted
I20240501 20:10:27.241540 10917 ts_tablet_manager.cc:1398] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240501 20:10:27.242384 4498 raft_consensus.cc:1261] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:10:27.243762 10940 consensus_queue.cc:1035] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:27.247167 10946 ts_tablet_manager.cc:1888] T 87c0a0f5a5094b818de6d87935cce7ce P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:27.247368 10946 log.cc:1196] T 87c0a0f5a5094b818de6d87935cce7ce P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/87c0a0f5a5094b818de6d87935cce7ce
I20240501 20:10:27.248754 7022 raft_consensus.cc:1261] T dfaa3794205d482c90aa4f860fa99cf3 P ef0a4ada548446b3a531cb39a3c4e38d [term 2 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:10:27.249608 10940 consensus_queue.cc:1035] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:27.261461 10946 ts_tablet_manager.cc:1909] T 87c0a0f5a5094b818de6d87935cce7ce P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting consensus metadata
I20240501 20:10:27.263219 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet 87c0a0f5a5094b818de6d87935cce7ce (table rebalancer_dynamic_table_new_name [id=be88a358b513433fa5e39f2e931bd198]) successfully deleted
I20240501 20:10:27.269487 10672 raft_consensus.cc:2932] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:27.281965 4091 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:27.286238 7019 raft_consensus.cc:2932] T 00ae88161b514e4aa4b8d134a19de001 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:27.288573 7281 consensus_queue.cc:237] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:27.291177 4499 raft_consensus.cc:1261] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:10:27.291265 7022 raft_consensus.cc:1261] T dfaa3794205d482c90aa4f860fa99cf3 P ef0a4ada548446b3a531cb39a3c4e38d [term 2 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:10:27.291918 10712 consensus_queue.cc:1035] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:10:27.294191 10940 consensus_queue.cc:1035] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:10:27.304525 7001 tablet_service.cc:1467] Processing CreateTablet for tablet 3371a9d68ee746aca0795e15686ba694 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e251cffa507f4bd2adc671356322485b]), partition=
I20240501 20:10:27.304936 7001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3371a9d68ee746aca0795e15686ba694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.306010 7134 tablet_service.cc:1467] Processing CreateTablet for tablet 3371a9d68ee746aca0795e15686ba694 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e251cffa507f4bd2adc671356322485b]), partition=
I20240501 20:10:27.306361 7134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3371a9d68ee746aca0795e15686ba694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.307546 7400 tablet_service.cc:1467] Processing CreateTablet for tablet 3371a9d68ee746aca0795e15686ba694 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e251cffa507f4bd2adc671356322485b]), partition=
I20240501 20:10:27.307906 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3371a9d68ee746aca0795e15686ba694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.309218 10931 tablet_bootstrap.cc:492] T d13178f10db0435891b97cebaed6dd5d P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:27.309561 10931 tablet_bootstrap.cc:492] T d13178f10db0435891b97cebaed6dd5d P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap complete.
I20240501 20:10:27.309803 10931 ts_tablet_manager.cc:1367] T d13178f10db0435891b97cebaed6dd5d P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.086s user 0.006s sys 0.003s
I20240501 20:10:27.313875 10712 raft_consensus.cc:2932] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:27.316851 4496 raft_consensus.cc:2932] T 00ae88161b514e4aa4b8d134a19de001 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:27.318532 4499 raft_consensus.cc:2932] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:27.321267 4077 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dfaa3794205d482c90aa4f860fa99cf3 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:27.318022 4091 catalog_manager.cc:5484] T 00ae88161b514e4aa4b8d134a19de001 P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:27.323621 10923 tablet_bootstrap.cc:492] T 3371a9d68ee746aca0795e15686ba694 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:27.325655 10923 tablet_bootstrap.cc:654] T 3371a9d68ee746aca0795e15686ba694 P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:27.310479 10931 raft_consensus.cc:348] T d13178f10db0435891b97cebaed6dd5d P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:27.326489 10931 raft_consensus.cc:729] T d13178f10db0435891b97cebaed6dd5d P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: LEARNER
I20240501 20:10:27.326738 10931 consensus_queue.cc:260] T d13178f10db0435891b97cebaed6dd5d P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:27.331373 10931 ts_tablet_manager.cc:1398] T d13178f10db0435891b97cebaed6dd5d P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.021s user 0.004s sys 0.000s
I20240501 20:10:27.339056 10960 tablet_bootstrap.cc:492] T 3371a9d68ee746aca0795e15686ba694 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:27.340682 10960 tablet_bootstrap.cc:654] T 3371a9d68ee746aca0795e15686ba694 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:27.342693 10923 tablet_bootstrap.cc:492] T 3371a9d68ee746aca0795e15686ba694 P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:27.343029 10923 ts_tablet_manager.cc:1367] T 3371a9d68ee746aca0795e15686ba694 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.002s
I20240501 20:10:27.344113 10923 raft_consensus.cc:348] T 3371a9d68ee746aca0795e15686ba694 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:27.344520 10923 raft_consensus.cc:374] T 3371a9d68ee746aca0795e15686ba694 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:27.344674 10923 raft_consensus.cc:729] T 3371a9d68ee746aca0795e15686ba694 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:27.345088 10923 consensus_queue.cc:260] T 3371a9d68ee746aca0795e15686ba694 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:27.345395 7022 raft_consensus.cc:2932] T dfaa3794205d482c90aa4f860fa99cf3 P ef0a4ada548446b3a531cb39a3c4e38d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:27.346855 4254 tablet_copy_service.cc:342] P dca57823003e4729b769ca12367d7530: Request end of tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-d13178f10db0435891b97cebaed6dd5d received from {username='slave'} at 127.29.219.67:51285
I20240501 20:10:27.346967 4254 tablet_copy_service.cc:433] P dca57823003e4729b769ca12367d7530: ending tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-d13178f10db0435891b97cebaed6dd5d on tablet d13178f10db0435891b97cebaed6dd5d with peer 3e974c03d3c546f9ab6c91e762a5426b
I20240501 20:10:27.338371 4090 catalog_manager.cc:5484] T dfaa3794205d482c90aa4f860fa99cf3 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 8 to 10, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New cstate: current_term: 2 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:27.362782 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet dfaa3794205d482c90aa4f860fa99cf3 with delete_type TABLET_DATA_TOMBSTONED (TS dca57823003e4729b769ca12367d7530 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:27.363132 10944 tablet_replica.cc:330] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:27.363301 10944 raft_consensus.cc:2229] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:27.363449 10944 raft_consensus.cc:2258] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:27.364710 10944 ts_tablet_manager.cc:1875] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:27.365675 10947 tablet_copy_client.cc:308] T c5b4fbeda51f40b8ac54baf5fb241710 P 10b47782558c41ee91f5e164a372049a: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.72:39817
I20240501 20:10:27.366497 7441 tablet_copy_service.cc:140] P b9957d4086bc4a76b18ed9265a5739d5: Received BeginTabletCopySession request for tablet c5b4fbeda51f40b8ac54baf5fb241710 from peer 10b47782558c41ee91f5e164a372049a ({username='slave'} at 127.29.219.66:55097)
I20240501 20:10:27.366631 10923 ts_tablet_manager.cc:1398] T 3371a9d68ee746aca0795e15686ba694 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240501 20:10:27.366665 7441 tablet_copy_service.cc:161] P b9957d4086bc4a76b18ed9265a5739d5: Beginning new tablet copy session on tablet c5b4fbeda51f40b8ac54baf5fb241710 from peer 10b47782558c41ee91f5e164a372049a at {username='slave'} at 127.29.219.66:55097: session id = 10b47782558c41ee91f5e164a372049a-c5b4fbeda51f40b8ac54baf5fb241710
I20240501 20:10:27.369896 7441 tablet_copy_source_session.cc:205] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:27.371596 10944 ts_tablet_manager.cc:1888] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 20:10:27.371866 10944 log.cc:1196] T dfaa3794205d482c90aa4f860fa99cf3 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/dfaa3794205d482c90aa4f860fa99cf3
I20240501 20:10:27.373056 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet dfaa3794205d482c90aa4f860fa99cf3 (table rebalance_test_table_15 [id=2478c1c300d14f2a9826930dc16a5dec]) successfully deleted
I20240501 20:10:27.388711 10871 raft_consensus.cc:482] T da878c45e7184273ba6d3497e9eef35d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:27.388760 10921 tablet_bootstrap.cc:492] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:27.388873 10871 raft_consensus.cc:504] T da878c45e7184273ba6d3497e9eef35d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } }
I20240501 20:10:27.389621 10871 leader_election.cc:288] T da878c45e7184273ba6d3497e9eef35d P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:27.389997 10947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b4fbeda51f40b8ac54baf5fb241710. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.390425 10921 tablet_bootstrap.cc:654] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:27.390515 4628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da878c45e7184273ba6d3497e9eef35d" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:27.391873 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da878c45e7184273ba6d3497e9eef35d" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" is_pre_election: true
I20240501 20:10:27.392238 7022 raft_consensus.cc:2379] T da878c45e7184273ba6d3497e9eef35d P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db0742ee3860468590eefef6ae6074fd in current term 1: Already voted for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in this term.
I20240501 20:10:27.393072 7090 leader_election.cc:302] T da878c45e7184273ba6d3497e9eef35d P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db0742ee3860468590eefef6ae6074fd; no voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:27.393496 10871 raft_consensus.cc:3037] T da878c45e7184273ba6d3497e9eef35d P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:27.397359 10871 raft_consensus.cc:2726] T da878c45e7184273ba6d3497e9eef35d P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:27.404870 10921 tablet_bootstrap.cc:492] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d: No bootstrap required, opened a new log
I20240501 20:10:27.405167 10921 ts_tablet_manager.cc:1367] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.007s
I20240501 20:10:27.407613 10921 raft_consensus.cc:348] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:27.407892 10921 raft_consensus.cc:374] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:27.408006 10921 raft_consensus.cc:729] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: FOLLOWER
I20240501 20:10:27.408368 10921 consensus_queue.cc:260] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:27.409332 10960 tablet_bootstrap.cc:492] T 3371a9d68ee746aca0795e15686ba694 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:27.410161 10948 tablet_copy_client.cc:308] T 797b2374d4214c8eb39982c339a7fbad P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.70:36321
I20240501 20:10:27.411104 7175 tablet_copy_service.cc:140] P db0742ee3860468590eefef6ae6074fd: Received BeginTabletCopySession request for tablet 797b2374d4214c8eb39982c339a7fbad from peer ef0a4ada548446b3a531cb39a3c4e38d ({username='slave'} at 127.29.219.69:39809)
I20240501 20:10:27.411271 7175 tablet_copy_service.cc:161] P db0742ee3860468590eefef6ae6074fd: Beginning new tablet copy session on tablet 797b2374d4214c8eb39982c339a7fbad from peer ef0a4ada548446b3a531cb39a3c4e38d at {username='slave'} at 127.29.219.69:39809: session id = ef0a4ada548446b3a531cb39a3c4e38d-797b2374d4214c8eb39982c339a7fbad
I20240501 20:10:27.412858 10958 raft_consensus.cc:482] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:27.413002 10958 raft_consensus.cc:504] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:27.413673 10958 leader_election.cc:288] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:27.413935 7175 tablet_copy_source_session.cc:205] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:27.414238 10921 ts_tablet_manager.cc:1398] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20240501 20:10:27.414369 4628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3bb8370e7c41dc9cf7f1c835ac275b" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:27.415026 4363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3bb8370e7c41dc9cf7f1c835ac275b" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:10:27.415232 4363 raft_consensus.cc:2454] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:27.415550 10948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 797b2374d4214c8eb39982c339a7fbad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.414599 4628 raft_consensus.cc:2454] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:27.416010 10960 ts_tablet_manager.cc:1367] T 3371a9d68ee746aca0795e15686ba694 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.077s user 0.000s sys 0.010s
I20240501 20:10:27.416080 4170 leader_election.cc:302] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:27.416805 10672 raft_consensus.cc:2781] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:27.416893 10672 raft_consensus.cc:482] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:27.416952 10672 raft_consensus.cc:3037] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:27.416886 10960 raft_consensus.cc:348] T 3371a9d68ee746aca0795e15686ba694 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:27.417124 10960 raft_consensus.cc:374] T 3371a9d68ee746aca0795e15686ba694 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:27.417222 10960 raft_consensus.cc:729] T 3371a9d68ee746aca0795e15686ba694 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:27.419543 10672 raft_consensus.cc:504] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:27.419890 10960 consensus_queue.cc:260] T 3371a9d68ee746aca0795e15686ba694 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:27.420544 10960 ts_tablet_manager.cc:1398] T 3371a9d68ee746aca0795e15686ba694 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 20:10:27.421906 10672 leader_election.cc:288] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:27.423193 4628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3bb8370e7c41dc9cf7f1c835ac275b" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:10:27.423421 4628 raft_consensus.cc:3037] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:27.426553 4628 raft_consensus.cc:2454] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:27.429124 4363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3bb8370e7c41dc9cf7f1c835ac275b" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:10:27.429378 4363 raft_consensus.cc:3037] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:27.432969 4363 raft_consensus.cc:2454] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:27.433743 4170 leader_election.cc:302] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:27.434185 10672 raft_consensus.cc:2781] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:27.434391 10672 raft_consensus.cc:686] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:10:27.435130 10672 consensus_queue.cc:237] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:27.445091 10973 ts_tablet_manager.cc:902] T 26ed0760e9074bd896e55e99756602bb P dca57823003e4729b769ca12367d7530: Initiating tablet copy from peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:27.448834 10948 tablet_copy_client.cc:783] T 797b2374d4214c8eb39982c339a7fbad P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:27.449170 10948 tablet_copy_client.cc:647] T 797b2374d4214c8eb39982c339a7fbad P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:27.452327 4089 catalog_manager.cc:5484] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:27.454411 10948 tablet_copy_client.cc:523] T 797b2374d4214c8eb39982c339a7fbad P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:27.455811 10701 raft_consensus.cc:482] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:27.457103 10701 raft_consensus.cc:504] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:27.457835 10701 leader_election.cc:288] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:27.458469 7152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3371a9d68ee746aca0795e15686ba694" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:27.458722 7152 raft_consensus.cc:2454] T 3371a9d68ee746aca0795e15686ba694 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:27.454309 10947 tablet_copy_client.cc:783] T c5b4fbeda51f40b8ac54baf5fb241710 P 10b47782558c41ee91f5e164a372049a: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:27.459106 7417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3371a9d68ee746aca0795e15686ba694" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:27.459323 7417 raft_consensus.cc:2454] T 3371a9d68ee746aca0795e15686ba694 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 0.
I20240501 20:10:27.459543 6955 leader_election.cc:302] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db0742ee3860468590eefef6ae6074fd, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:27.460089 10700 raft_consensus.cc:2781] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:27.460196 10700 raft_consensus.cc:482] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:27.460268 10700 raft_consensus.cc:3037] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:27.462185 10973 tablet_copy_client.cc:308] T 26ed0760e9074bd896e55e99756602bb P dca57823003e4729b769ca12367d7530: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.66:40689
I20240501 20:10:27.462947 4388 tablet_copy_service.cc:140] P 10b47782558c41ee91f5e164a372049a: Received BeginTabletCopySession request for tablet 26ed0760e9074bd896e55e99756602bb from peer dca57823003e4729b769ca12367d7530 ({username='slave'} at 127.29.219.65:41491)
I20240501 20:10:27.463097 4388 tablet_copy_service.cc:161] P 10b47782558c41ee91f5e164a372049a: Beginning new tablet copy session on tablet 26ed0760e9074bd896e55e99756602bb from peer dca57823003e4729b769ca12367d7530 at {username='slave'} at 127.29.219.65:41491: session id = dca57823003e4729b769ca12367d7530-26ed0760e9074bd896e55e99756602bb
I20240501 20:10:27.463721 10947 tablet_copy_client.cc:647] T c5b4fbeda51f40b8ac54baf5fb241710 P 10b47782558c41ee91f5e164a372049a: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:27.465900 4388 tablet_copy_source_session.cc:205] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:27.467124 4498 raft_consensus.cc:1203] T d13178f10db0435891b97cebaed6dd5d P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:27.467680 10973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26ed0760e9074bd896e55e99756602bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.468043 10700 raft_consensus.cc:504] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:27.469478 7152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3371a9d68ee746aca0795e15686ba694" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:27.469748 7152 raft_consensus.cc:3037] T 3371a9d68ee746aca0795e15686ba694 P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:27.471115 10700 leader_election.cc:288] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Requested vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:27.472496 7152 raft_consensus.cc:2454] T 3371a9d68ee746aca0795e15686ba694 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:27.472986 6955 leader_election.cc:302] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db0742ee3860468590eefef6ae6074fd, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:27.473495 10700 raft_consensus.cc:2781] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:27.473637 10700 raft_consensus.cc:686] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Becoming Leader. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Running, Role: LEADER
I20240501 20:10:27.473913 10700 consensus_queue.cc:237] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:27.474165 10947 tablet_copy_client.cc:523] T c5b4fbeda51f40b8ac54baf5fb241710 P 10b47782558c41ee91f5e164a372049a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:27.474606 7417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3371a9d68ee746aca0795e15686ba694" candidate_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:27.474849 7417 raft_consensus.cc:3037] T 3371a9d68ee746aca0795e15686ba694 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:27.477722 10947 tablet_bootstrap.cc:492] T c5b4fbeda51f40b8ac54baf5fb241710 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:27.480670 10973 tablet_copy_client.cc:783] T 26ed0760e9074bd896e55e99756602bb P dca57823003e4729b769ca12367d7530: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:27.480937 10973 tablet_copy_client.cc:647] T 26ed0760e9074bd896e55e99756602bb P dca57823003e4729b769ca12367d7530: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:27.485491 10948 tablet_bootstrap.cc:492] T 797b2374d4214c8eb39982c339a7fbad P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:27.488191 7417 raft_consensus.cc:2454] T 3371a9d68ee746aca0795e15686ba694 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ef0a4ada548446b3a531cb39a3c4e38d in term 1.
I20240501 20:10:27.502480 10948 tablet_bootstrap.cc:492] T 797b2374d4214c8eb39982c339a7fbad P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:27.504993 10948 tablet_bootstrap.cc:492] T 797b2374d4214c8eb39982c339a7fbad P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap complete.
I20240501 20:10:27.507807 10948 ts_tablet_manager.cc:1367] T 797b2374d4214c8eb39982c339a7fbad P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.000s
I20240501 20:10:27.509131 10948 raft_consensus.cc:348] T 797b2374d4214c8eb39982c339a7fbad P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:27.509362 10948 raft_consensus.cc:729] T 797b2374d4214c8eb39982c339a7fbad P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: LEARNER
I20240501 20:10:27.509567 10948 consensus_queue.cc:260] T 797b2374d4214c8eb39982c339a7fbad P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:27.514425 10948 ts_tablet_manager.cc:1398] T 797b2374d4214c8eb39982c339a7fbad P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 20:10:27.515250 10973 tablet_copy_client.cc:523] T 26ed0760e9074bd896e55e99756602bb P dca57823003e4729b769ca12367d7530: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:27.515378 4628 consensus_queue.cc:237] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:27.516255 7175 tablet_copy_service.cc:342] P db0742ee3860468590eefef6ae6074fd: Request end of tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-797b2374d4214c8eb39982c339a7fbad received from {username='slave'} at 127.29.219.69:39809
I20240501 20:10:27.516383 7175 tablet_copy_service.cc:433] P db0742ee3860468590eefef6ae6074fd: ending tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-797b2374d4214c8eb39982c339a7fbad on tablet 797b2374d4214c8eb39982c339a7fbad with peer ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:27.517897 4090 catalog_manager.cc:5484] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: term changed from 0 to 1, leader changed from <none> to ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69). New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:27.519393 10973 tablet_bootstrap.cc:492] T 26ed0760e9074bd896e55e99756602bb P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:27.520395 10645 raft_consensus.cc:482] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:27.520540 10645 raft_consensus.cc:504] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:27.521214 10645 leader_election.cc:288] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:27.521836 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced3d988d474a8e8ee8f0a1fb4cc88e" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:27.521847 7281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced3d988d474a8e8ee8f0a1fb4cc88e" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:27.522053 4627 raft_consensus.cc:2454] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:10:27.522064 7281 raft_consensus.cc:2454] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 0.
I20240501 20:10:27.522475 4433 leader_election.cc:302] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:27.523139 10645 raft_consensus.cc:2781] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:27.523236 10645 raft_consensus.cc:482] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:27.523325 10645 raft_consensus.cc:3037] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 3e974c03d3c546f9ab6c91e762a5426b [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:27.526149 10645 raft_consensus.cc:504] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:27.527287 10645 leader_election.cc:288] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Requested vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:27.528028 7283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced3d988d474a8e8ee8f0a1fb4cc88e" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:27.528223 7283 raft_consensus.cc:3037] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:27.528805 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ced3d988d474a8e8ee8f0a1fb4cc88e" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:10:27.529086 4627 raft_consensus.cc:3037] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:27.530673 7283 raft_consensus.cc:2454] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:10:27.532586 4433 leader_election.cc:302] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:27.532863 10645 raft_consensus.cc:2781] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:27.532996 10645 raft_consensus.cc:686] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:10:27.533241 10645 consensus_queue.cc:237] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:27.535732 4627 raft_consensus.cc:2454] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:10:27.537834 4090 catalog_manager.cc:5484] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } }
W20240501 20:10:27.547675 4562 consensus_peers.cc:485] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Couldn't send request to peer db0742ee3860468590eefef6ae6074fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65b0048212a745c08ab24ead346bf74e. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:27.548460 7283 raft_consensus.cc:1261] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:10:27.549122 10377 consensus_queue.cc:1035] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:27.551128 4363 raft_consensus.cc:1261] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a [term 2 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 20:10:27.551956 10377 consensus_queue.cc:1035] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:27.560835 10947 tablet_bootstrap.cc:492] T c5b4fbeda51f40b8ac54baf5fb241710 P 10b47782558c41ee91f5e164a372049a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:27.561290 10947 tablet_bootstrap.cc:492] T c5b4fbeda51f40b8ac54baf5fb241710 P 10b47782558c41ee91f5e164a372049a: Bootstrap complete.
I20240501 20:10:27.561885 10947 ts_tablet_manager.cc:1367] T c5b4fbeda51f40b8ac54baf5fb241710 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.084s user 0.009s sys 0.005s
I20240501 20:10:27.562862 10947 raft_consensus.cc:348] T c5b4fbeda51f40b8ac54baf5fb241710 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:27.563177 10947 raft_consensus.cc:729] T c5b4fbeda51f40b8ac54baf5fb241710 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: LEARNER
I20240501 20:10:27.563364 10947 consensus_queue.cc:260] T c5b4fbeda51f40b8ac54baf5fb241710 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:27.564792 10947 ts_tablet_manager.cc:1398] T c5b4fbeda51f40b8ac54baf5fb241710 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:27.565459 7441 tablet_copy_service.cc:342] P b9957d4086bc4a76b18ed9265a5739d5: Request end of tablet copy session 10b47782558c41ee91f5e164a372049a-c5b4fbeda51f40b8ac54baf5fb241710 received from {username='slave'} at 127.29.219.66:55097
I20240501 20:10:27.565618 7441 tablet_copy_service.cc:433] P b9957d4086bc4a76b18ed9265a5739d5: ending tablet copy session 10b47782558c41ee91f5e164a372049a-c5b4fbeda51f40b8ac54baf5fb241710 on tablet c5b4fbeda51f40b8ac54baf5fb241710 with peer 10b47782558c41ee91f5e164a372049a
I20240501 20:10:27.569929 10377 raft_consensus.cc:2932] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:27.570451 4365 raft_consensus.cc:2932] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:27.573274 7283 raft_consensus.cc:2932] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:27.576148 10973 tablet_bootstrap.cc:492] T 26ed0760e9074bd896e55e99756602bb P dca57823003e4729b769ca12367d7530: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:27.576615 10973 tablet_bootstrap.cc:492] T 26ed0760e9074bd896e55e99756602bb P dca57823003e4729b769ca12367d7530: Bootstrap complete.
I20240501 20:10:27.576923 10973 ts_tablet_manager.cc:1367] T 26ed0760e9074bd896e55e99756602bb P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.058s user 0.015s sys 0.005s
I20240501 20:10:27.577718 10973 raft_consensus.cc:348] T 26ed0760e9074bd896e55e99756602bb P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:27.578006 10973 raft_consensus.cc:729] T 26ed0760e9074bd896e55e99756602bb P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: LEARNER
I20240501 20:10:27.578199 10973 consensus_queue.cc:260] T 26ed0760e9074bd896e55e99756602bb P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:27.578770 10973 ts_tablet_manager.cc:1398] T 26ed0760e9074bd896e55e99756602bb P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:10:27.578930 4090 catalog_manager.cc:5484] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 7 to 8, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New cstate: current_term: 2 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:27.581542 4388 tablet_copy_service.cc:342] P 10b47782558c41ee91f5e164a372049a: Request end of tablet copy session dca57823003e4729b769ca12367d7530-26ed0760e9074bd896e55e99756602bb received from {username='slave'} at 127.29.219.65:41491
I20240501 20:10:27.581694 4388 tablet_copy_service.cc:433] P 10b47782558c41ee91f5e164a372049a: ending tablet copy session dca57823003e4729b769ca12367d7530-26ed0760e9074bd896e55e99756602bb on tablet 26ed0760e9074bd896e55e99756602bb with peer dca57823003e4729b769ca12367d7530
I20240501 20:10:27.582193 10931 ts_tablet_manager.cc:902] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b: Initiating tablet copy from peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:27.622566 10377 consensus_queue.cc:1035] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:27.644982 4089 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:27.653787 7420 raft_consensus.cc:1261] T 3371a9d68ee746aca0795e15686ba694 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:27.654981 10700 consensus_queue.cc:1035] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:27.666960 10377 consensus_queue.cc:1035] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:27.689750 7153 raft_consensus.cc:1261] T 3371a9d68ee746aca0795e15686ba694 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:27.690881 10700 consensus_queue.cc:1035] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:27.705507 10931 tablet_copy_client.cc:308] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.69:46237
I20240501 20:10:27.706224 7042 tablet_copy_service.cc:140] P ef0a4ada548446b3a531cb39a3c4e38d: Received BeginTabletCopySession request for tablet 48528ec242974de4ac20eb0161f0cb05 from peer 3e974c03d3c546f9ab6c91e762a5426b ({username='slave'} at 127.29.219.67:45609)
I20240501 20:10:27.706462 7042 tablet_copy_service.cc:161] P ef0a4ada548446b3a531cb39a3c4e38d: Beginning new tablet copy session on tablet 48528ec242974de4ac20eb0161f0cb05 from peer 3e974c03d3c546f9ab6c91e762a5426b at {username='slave'} at 127.29.219.67:45609: session id = 3e974c03d3c546f9ab6c91e762a5426b-48528ec242974de4ac20eb0161f0cb05
I20240501 20:10:27.710250 7042 tablet_copy_source_session.cc:205] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:27.713027 10931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48528ec242974de4ac20eb0161f0cb05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.719358 11009 tablet.cc:1722] T 3371a9d68ee746aca0795e15686ba694 P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:27.721336 4363 raft_consensus.cc:1203] T c5b4fbeda51f40b8ac54baf5fb241710 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:27.723131 11009 tablet.cc:1620] T 3371a9d68ee746aca0795e15686ba694 P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:10:27.724259 11007 tablet.cc:1722] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:27.724835 11007 tablet.cc:1620] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:27.727909 11005 ts_tablet_manager.cc:902] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Initiating tablet copy from peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:27.734774 4089 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=e251cffa507f4bd2adc671356322485b] alter complete (version 1)
I20240501 20:10:27.739327 4084 catalog_manager.cc:5576] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817) does not have the latest schema for tablet 3371a9d68ee746aca0795e15686ba694 (table rebalancer_dynamic_table [id=e251cffa507f4bd2adc671356322485b]). Expected version 1 got 0
I20240501 20:10:27.742930 7152 consensus_queue.cc:237] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:27.760219 11012 tablet.cc:1722] T 3371a9d68ee746aca0795e15686ba694 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:27.764729 4084 catalog_manager.cc:5576] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817) does not have the latest schema for tablet 3371a9d68ee746aca0795e15686ba694 (table rebalancer_dynamic_table [id=e251cffa507f4bd2adc671356322485b]). Expected version 1 got 0
I20240501 20:10:27.765923 4234 raft_consensus.cc:1261] T 5215e3f2040747f29aa1c9181a628096 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:27.767530 11012 tablet.cc:1620] T 3371a9d68ee746aca0795e15686ba694 P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
W20240501 20:10:27.768090 7090 consensus_peers.cc:485] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5215e3f2040747f29aa1c9181a628096. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:27.769027 7283 raft_consensus.cc:1261] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:27.788100 10931 tablet_copy_client.cc:783] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:27.788534 10931 tablet_copy_client.cc:647] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:27.794000 10648 consensus_queue.cc:1035] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:27.802695 10931 tablet_copy_client.cc:523] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:27.809762 10931 tablet_bootstrap.cc:492] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:27.812026 7022 raft_consensus.cc:1203] T 797b2374d4214c8eb39982c339a7fbad P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:27.817402 11015 consensus_queue.cc:1035] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:27.823819 11014 raft_consensus.cc:2932] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:27.831028 4234 raft_consensus.cc:2932] T 5215e3f2040747f29aa1c9181a628096 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:27.835855 4084 catalog_manager.cc:5484] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 4 to 5, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:27.844907 4084 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 20:10:27.857412 11007 tablet.cc:1722] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:27.857929 11007 tablet.cc:1620] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d: MemRowSet was empty: no flush needed.
I20240501 20:10:27.864521 11012 tablet.cc:1722] T 3371a9d68ee746aca0795e15686ba694 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:27.865025 11012 tablet.cc:1620] T 3371a9d68ee746aca0795e15686ba694 P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:27.865073 11009 tablet.cc:1722] T 3371a9d68ee746aca0795e15686ba694 P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 20:10:27.865506 11009 tablet.cc:1620] T 3371a9d68ee746aca0795e15686ba694 P db0742ee3860468590eefef6ae6074fd: MemRowSet was empty: no flush needed.
I20240501 20:10:27.867534 4234 raft_consensus.cc:1203] T 26ed0760e9074bd896e55e99756602bb P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:27.878161 4089 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=e251cffa507f4bd2adc671356322485b] alter complete (version 2)
I20240501 20:10:27.882853 10931 tablet_bootstrap.cc:492] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:27.883249 10931 tablet_bootstrap.cc:492] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap complete.
I20240501 20:10:27.883494 10931 ts_tablet_manager.cc:1367] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.074s user 0.003s sys 0.005s
I20240501 20:10:27.884202 10931 raft_consensus.cc:348] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:27.884426 10931 raft_consensus.cc:729] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: LEARNER
I20240501 20:10:27.884593 10931 consensus_queue.cc:260] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:27.898226 7283 raft_consensus.cc:2932] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:27.904440 10958 consensus_queue.cc:1035] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:27.908541 10931 ts_tablet_manager.cc:1398] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20240501 20:10:27.909826 7042 tablet_copy_service.cc:342] P ef0a4ada548446b3a531cb39a3c4e38d: Request end of tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-48528ec242974de4ac20eb0161f0cb05 received from {username='slave'} at 127.29.219.67:45609
I20240501 20:10:27.909977 7042 tablet_copy_service.cc:433] P ef0a4ada548446b3a531cb39a3c4e38d: ending tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-48528ec242974de4ac20eb0161f0cb05 on tablet 48528ec242974de4ac20eb0161f0cb05 with peer 3e974c03d3c546f9ab6c91e762a5426b
I20240501 20:10:27.927903 11005 tablet_copy_client.cc:308] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.65:42941
I20240501 20:10:27.928761 4254 tablet_copy_service.cc:140] P dca57823003e4729b769ca12367d7530: Received BeginTabletCopySession request for tablet 00ae88161b514e4aa4b8d134a19de001 from peer 1d34e18a1d394d3f91c9f50d1c7bea4e ({username='slave'} at 127.29.219.68:58413)
I20240501 20:10:27.928944 4254 tablet_copy_service.cc:161] P dca57823003e4729b769ca12367d7530: Beginning new tablet copy session on tablet 00ae88161b514e4aa4b8d134a19de001 from peer 1d34e18a1d394d3f91c9f50d1c7bea4e at {username='slave'} at 127.29.219.68:58413: session id = 1d34e18a1d394d3f91c9f50d1c7bea4e-00ae88161b514e4aa4b8d134a19de001
I20240501 20:10:27.932534 4254 tablet_copy_source_session.cc:205] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:27.934439 11005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ae88161b514e4aa4b8d134a19de001. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:27.939718 10958 consensus_queue.cc:1035] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:27.941357 11005 tablet_copy_client.cc:783] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:27.942009 11005 tablet_copy_client.cc:647] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:27.943827 11005 tablet_copy_client.cc:523] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:27.947607 11005 tablet_bootstrap.cc:492] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:27.973049 11005 tablet_bootstrap.cc:492] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:27.976218 11005 tablet_bootstrap.cc:492] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap complete.
I20240501 20:10:27.976521 11005 ts_tablet_manager.cc:1367] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.010s
I20240501 20:10:27.977304 11005 raft_consensus.cc:348] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:27.977546 11005 raft_consensus.cc:729] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: LEARNER
I20240501 20:10:27.977782 11005 consensus_queue.cc:260] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:27.978411 11005 ts_tablet_manager.cc:1398] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:10:27.979249 4254 tablet_copy_service.cc:342] P dca57823003e4729b769ca12367d7530: Request end of tablet copy session 1d34e18a1d394d3f91c9f50d1c7bea4e-00ae88161b514e4aa4b8d134a19de001 received from {username='slave'} at 127.29.219.68:58413
I20240501 20:10:27.979352 4254 tablet_copy_service.cc:433] P dca57823003e4729b769ca12367d7530: ending tablet copy session 1d34e18a1d394d3f91c9f50d1c7bea4e-00ae88161b514e4aa4b8d134a19de001 on tablet 00ae88161b514e4aa4b8d134a19de001 with peer 1d34e18a1d394d3f91c9f50d1c7bea4e
I20240501 20:10:27.979983 10672 raft_consensus.cc:1050] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530: attempting to promote NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b to VOTER
I20240501 20:10:27.980597 4498 consensus_queue.cc:237] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:27.980787 10672 consensus_queue.cc:237] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:27.986661 4496 raft_consensus.cc:1203] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:27.990604 4090 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 20:10:27.996752 10649 consensus_queue.cc:1035] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:27.997010 7283 raft_consensus.cc:1261] T d13178f10db0435891b97cebaed6dd5d P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:27.997525 10958 consensus_queue.cc:1035] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:28.005326 11007 tablet.cc:1722] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240501 20:10:28.012008 4432 consensus_peers.cc:485] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b -> Peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Couldn't send request to peer db0742ee3860468590eefef6ae6074fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca4f195a6dd14be981395538c2a33ad9. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:28.012085 11012 tablet.cc:1722] T 3371a9d68ee746aca0795e15686ba694 P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:28.016608 7281 raft_consensus.cc:1261] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:28.017838 10649 consensus_queue.cc:1035] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:28.018345 4234 raft_consensus.cc:1261] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:28.023996 10649 consensus_queue.cc:1035] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:28.024374 4090 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table_new_name [id=e251cffa507f4bd2adc671356322485b] alter complete (version 3)
I20240501 20:10:28.027707 11041 consensus_queue.cc:1035] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:28.030254 11009 tablet.cc:1722] T 3371a9d68ee746aca0795e15686ba694 P db0742ee3860468590eefef6ae6074fd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 20:10:28.050346 10649 raft_consensus.cc:2932] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:28.051119 4498 raft_consensus.cc:1261] T d13178f10db0435891b97cebaed6dd5d P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:28.051398 7420 raft_consensus.cc:1261] T d13178f10db0435891b97cebaed6dd5d P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:28.051776 10958 consensus_queue.cc:1035] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:28.052700 11034 consensus_queue.cc:1035] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:28.067500 4234 raft_consensus.cc:2932] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:28.093955 4089 catalog_manager.cc:5484] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 6 to 7, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:28.111733 10958 raft_consensus.cc:2932] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:28.120626 4090 catalog_manager.cc:5484] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:28.123045 7420 raft_consensus.cc:2932] T d13178f10db0435891b97cebaed6dd5d P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
W20240501 20:10:28.128609 4433 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.71:39447, user_credentials={real_user=slave}} blocked reactor thread for 59957.3us
I20240501 20:10:28.130077 4089 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:28.130581 4089 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 20:10:28.117653 7287 raft_consensus.cc:2932] T d13178f10db0435891b97cebaed6dd5d P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:28.141098 7283 raft_consensus.cc:2932] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:28.155644 4084 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:28.181800 11051 ts_tablet_manager.cc:902] T 65b0048212a745c08ab24ead346bf74e P db0742ee3860468590eefef6ae6074fd: Initiating tablet copy from peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:28.183748 7134 tablet_service.cc:1467] Processing CreateTablet for tablet 67703d84deb9417bb21bde6365deda2f (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=4966f0b362754500952a1e7f97c65b0b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:28.183912 4611 tablet_service.cc:1467] Processing CreateTablet for tablet 67703d84deb9417bb21bde6365deda2f (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=4966f0b362754500952a1e7f97c65b0b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:28.184254 7134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67703d84deb9417bb21bde6365deda2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:28.184327 4611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67703d84deb9417bb21bde6365deda2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:28.185159 4498 raft_consensus.cc:2932] T d13178f10db0435891b97cebaed6dd5d P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:28.186571 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 3ffe6b742c3743d8b7de7a0c4105637f (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=4966f0b362754500952a1e7f97c65b0b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:28.186936 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ffe6b742c3743d8b7de7a0c4105637f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:28.189201 4215 tablet_service.cc:1467] Processing CreateTablet for tablet bc8e3edd983e4be9a9726e73f04f08e3 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=4966f0b362754500952a1e7f97c65b0b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:28.189613 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc8e3edd983e4be9a9726e73f04f08e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:28.189601 7400 tablet_service.cc:1467] Processing CreateTablet for tablet 67703d84deb9417bb21bde6365deda2f (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=4966f0b362754500952a1e7f97c65b0b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:28.190062 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67703d84deb9417bb21bde6365deda2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:28.190984 7401 tablet_service.cc:1467] Processing CreateTablet for tablet bc8e3edd983e4be9a9726e73f04f08e3 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=4966f0b362754500952a1e7f97c65b0b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:28.191356 7401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc8e3edd983e4be9a9726e73f04f08e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:28.197536 4628 raft_consensus.cc:1203] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:28.201109 4346 tablet_service.cc:1467] Processing CreateTablet for tablet bc8e3edd983e4be9a9726e73f04f08e3 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=4966f0b362754500952a1e7f97c65b0b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:28.201402 7133 tablet_service.cc:1467] Processing CreateTablet for tablet 3ffe6b742c3743d8b7de7a0c4105637f (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=4966f0b362754500952a1e7f97c65b0b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:28.201558 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc8e3edd983e4be9a9726e73f04f08e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:28.201836 7133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ffe6b742c3743d8b7de7a0c4105637f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:28.203009 4213 tablet_service.cc:1467] Processing CreateTablet for tablet 3ffe6b742c3743d8b7de7a0c4105637f (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=4966f0b362754500952a1e7f97c65b0b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:28.203380 4213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ffe6b742c3743d8b7de7a0c4105637f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:28.205683 4234 consensus_queue.cc:237] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:28.207655 4089 catalog_manager.cc:5771] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: Sending DeleteTablet for 3 replicas of tablet 3371a9d68ee746aca0795e15686ba694
I20240501 20:10:28.208360 7001 tablet_service.cc:1514] Processing DeleteTablet for tablet 3371a9d68ee746aca0795e15686ba694 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:28 UTC) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:28.209163 7135 tablet_service.cc:1514] Processing DeleteTablet for tablet 3371a9d68ee746aca0795e15686ba694 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:28 UTC) from {username='slave'} at 127.0.0.1:52036
I20240501 20:10:28.212361 7399 tablet_service.cc:1514] Processing DeleteTablet for tablet 3371a9d68ee746aca0795e15686ba694 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 20:10:28 UTC) from {username='slave'} at 127.0.0.1:40182
I20240501 20:10:28.217202 7283 raft_consensus.cc:1261] T d13178f10db0435891b97cebaed6dd5d P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:28.219043 10958 consensus_queue.cc:1035] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:28.219604 4496 raft_consensus.cc:1261] T d13178f10db0435891b97cebaed6dd5d P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:28.220686 11034 consensus_queue.cc:1035] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:28.229121 11034 raft_consensus.cc:2932] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:28.232982 11062 tablet_replica.cc:330] T 3371a9d68ee746aca0795e15686ba694 P b9957d4086bc4a76b18ed9265a5739d5: stopping tablet replica
I20240501 20:10:28.233888 11062 raft_consensus.cc:2229] T 3371a9d68ee746aca0795e15686ba694 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:28.235137 11062 raft_consensus.cc:2258] T 3371a9d68ee746aca0795e15686ba694 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:28.233304 11059 tablet_bootstrap.cc:492] T bc8e3edd983e4be9a9726e73f04f08e3 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:28.234355 7287 raft_consensus.cc:2932] T d13178f10db0435891b97cebaed6dd5d P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:28.242569 4498 raft_consensus.cc:2932] T d13178f10db0435891b97cebaed6dd5d P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:28.243110 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d13178f10db0435891b97cebaed6dd5d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:28.248389 10931 ts_tablet_manager.cc:902] T 5215e3f2040747f29aa1c9181a628096 P 3e974c03d3c546f9ab6c91e762a5426b: Initiating tablet copy from peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:28.248356 4090 catalog_manager.cc:5484] T d13178f10db0435891b97cebaed6dd5d P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:28.258924 7401 tablet_service.cc:1514] Processing DeleteTablet for tablet d13178f10db0435891b97cebaed6dd5d with delete_type TABLET_DATA_TOMBSTONED (TS b9957d4086bc4a76b18ed9265a5739d5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40182
I20240501 20:10:28.260354 11059 tablet_bootstrap.cc:654] T bc8e3edd983e4be9a9726e73f04f08e3 P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:28.261719 11058 tablet_bootstrap.cc:492] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:28.263415 11058 tablet_bootstrap.cc:654] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:28.271279 11062 ts_tablet_manager.cc:1875] T 3371a9d68ee746aca0795e15686ba694 P b9957d4086bc4a76b18ed9265a5739d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:28.276834 11057 tablet_bootstrap.cc:492] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:28.277694 11059 tablet_bootstrap.cc:492] T bc8e3edd983e4be9a9726e73f04f08e3 P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:28.277849 11059 ts_tablet_manager.cc:1367] T bc8e3edd983e4be9a9726e73f04f08e3 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.000s
I20240501 20:10:28.278662 11057 tablet_bootstrap.cc:654] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:28.278647 11059 raft_consensus.cc:348] T bc8e3edd983e4be9a9726e73f04f08e3 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:28.278867 11059 raft_consensus.cc:374] T bc8e3edd983e4be9a9726e73f04f08e3 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:28.287386 11058 tablet_bootstrap.cc:492] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:28.287566 11058 ts_tablet_manager.cc:1367] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240501 20:10:28.291404 11059 raft_consensus.cc:729] T bc8e3edd983e4be9a9726e73f04f08e3 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:28.288303 11058 raft_consensus.cc:348] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:28.291877 11059 consensus_queue.cc:260] T bc8e3edd983e4be9a9726e73f04f08e3 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:28.293491 11058 raft_consensus.cc:374] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:28.293602 11058 raft_consensus.cc:729] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:28.294886 11059 ts_tablet_manager.cc:1398] T bc8e3edd983e4be9a9726e73f04f08e3 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.017s user 0.000s sys 0.003s
I20240501 20:10:28.298580 4498 consensus_queue.cc:237] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:28.299211 11058 consensus_queue.cc:260] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:28.306216 11059 tablet_bootstrap.cc:492] T 67703d84deb9417bb21bde6365deda2f P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:28.306980 11014 raft_consensus.cc:1050] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd: attempting to promote NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d to VOTER
I20240501 20:10:28.307683 11058 ts_tablet_manager.cc:1398] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.020s user 0.003s sys 0.000s
I20240501 20:10:28.307868 11014 consensus_queue.cc:237] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:28.309139 11061 tablet_replica.cc:330] T 3371a9d68ee746aca0795e15686ba694 P db0742ee3860468590eefef6ae6074fd: stopping tablet replica
I20240501 20:10:28.311825 11060 tablet_replica.cc:330] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d: stopping tablet replica
I20240501 20:10:28.312106 11060 raft_consensus.cc:2229] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Raft consensus shutting down.
I20240501 20:10:28.312584 11060 raft_consensus.cc:2258] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:28.310820 4088 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44422:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 20:10:28.313608 11060 ts_tablet_manager.cc:1875] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:28.313835 11059 tablet_bootstrap.cc:654] T 67703d84deb9417bb21bde6365deda2f P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:28.316040 11065 tablet_bootstrap.cc:492] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:28.315377 7420 raft_consensus.cc:1261] T 797b2374d4214c8eb39982c339a7fbad P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:28.318053 11065 tablet_bootstrap.cc:654] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:28.320221 4363 raft_consensus.cc:1261] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:28.320793 11048 consensus_queue.cc:1035] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 20:10:28.322444 4434 consensus_peers.cc:485] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2aa1a5f266e840698f2127289628dcba. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:28.325856 7420 raft_consensus.cc:1261] T 2aa1a5f266e840698f2127289628dcba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:28.326354 11048 consensus_queue.cc:1035] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:28.329185 7267 tablet_service.cc:1467] Processing CreateTablet for tablet 5f25c23e11dc48ee9d9496c1ae45fcaa (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ec12d35b5fd7479a8edeb64964c33483]), partition=
I20240501 20:10:28.329547 7267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f25c23e11dc48ee9d9496c1ae45fcaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:28.331200 4215 tablet_service.cc:1467] Processing CreateTablet for tablet 5f25c23e11dc48ee9d9496c1ae45fcaa (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ec12d35b5fd7479a8edeb64964c33483]), partition=
I20240501 20:10:28.331521 4215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f25c23e11dc48ee9d9496c1ae45fcaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:28.333537 10645 raft_consensus.cc:2932] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:28.337510 4365 raft_consensus.cc:2932] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:28.339180 7400 tablet_service.cc:1467] Processing CreateTablet for tablet 5f25c23e11dc48ee9d9496c1ae45fcaa (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ec12d35b5fd7479a8edeb64964c33483]), partition=
I20240501 20:10:28.339639 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f25c23e11dc48ee9d9496c1ae45fcaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:28.339900 11059 tablet_bootstrap.cc:492] T 67703d84deb9417bb21bde6365deda2f P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:28.340088 11059 ts_tablet_manager.cc:1367] T 67703d84deb9417bb21bde6365deda2f P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.000s
I20240501 20:10:28.341086 11059 raft_consensus.cc:348] T 67703d84deb9417bb21bde6365deda2f P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:28.341333 11059 raft_consensus.cc:374] T 67703d84deb9417bb21bde6365deda2f P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:28.341411 11059 raft_consensus.cc:729] T 67703d84deb9417bb21bde6365deda2f P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:28.341683 11059 consensus_queue.cc:260] T 67703d84deb9417bb21bde6365deda2f P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:28.353981 4090 catalog_manager.cc:5484] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 6 to 7, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:28.363345 11057 tablet_bootstrap.cc:492] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:28.364501 11077 raft_consensus.cc:1050] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5: attempting to promote NON_VOTER 10b47782558c41ee91f5e164a372049a to VOTER
I20240501 20:10:28.364606 11057 ts_tablet_manager.cc:1367] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.088s user 0.006s sys 0.001s
I20240501 20:10:28.365703 11057 raft_consensus.cc:348] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:28.366173 11077 consensus_queue.cc:237] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } }
I20240501 20:10:28.366991 11057 raft_consensus.cc:374] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:28.368693 11059 ts_tablet_manager.cc:1398] T 67703d84deb9417bb21bde6365deda2f P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20240501 20:10:28.368855 11065 tablet_bootstrap.cc:492] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:28.369048 11065 ts_tablet_manager.cc:1367] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.000s
I20240501 20:10:28.369793 11065 raft_consensus.cc:348] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:28.370303 11065 raft_consensus.cc:374] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:28.370414 11065 raft_consensus.cc:729] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:28.373433 11059 tablet_bootstrap.cc:492] T 5f25c23e11dc48ee9d9496c1ae45fcaa P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:28.375252 11059 tablet_bootstrap.cc:654] T 5f25c23e11dc48ee9d9496c1ae45fcaa P b9957d4086bc4a76b18ed9265a5739d5: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:28.367278 11057 raft_consensus.cc:729] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:28.378844 11057 consensus_queue.cc:260] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:28.380043 11057 ts_tablet_manager.cc:1398] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240501 20:10:28.380983 11057 tablet_bootstrap.cc:492] T 3ffe6b742c3743d8b7de7a0c4105637f P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:28.383399 11061 raft_consensus.cc:2229] T 3371a9d68ee746aca0795e15686ba694 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:28.383805 11065 consensus_queue.cc:260] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:28.384483 7287 raft_consensus.cc:1261] T 797b2374d4214c8eb39982c339a7fbad P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:28.384888 11065 ts_tablet_manager.cc:1398] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20240501 20:10:28.384898 7022 raft_consensus.cc:1261] T 797b2374d4214c8eb39982c339a7fbad P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:28.385355 11065 tablet_bootstrap.cc:492] T 67703d84deb9417bb21bde6365deda2f P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:28.386693 10871 consensus_queue.cc:1035] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.069s
I20240501 20:10:28.387909 10885 raft_consensus.cc:1050] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a: attempting to promote NON_VOTER dca57823003e4729b769ca12367d7530 to VOTER
I20240501 20:10:28.388693 11058 tablet_bootstrap.cc:492] T 3ffe6b742c3743d8b7de7a0c4105637f P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:28.388779 11062 ts_tablet_manager.cc:1888] T 3371a9d68ee746aca0795e15686ba694 P b9957d4086bc4a76b18ed9265a5739d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:28.388712 10885 consensus_queue.cc:237] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:28.388924 11062 log.cc:1196] T 3371a9d68ee746aca0795e15686ba694 P b9957d4086bc4a76b18ed9265a5739d5: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/wals/3371a9d68ee746aca0795e15686ba694
I20240501 20:10:28.399580 11058 tablet_bootstrap.cc:654] T 3ffe6b742c3743d8b7de7a0c4105637f P db0742ee3860468590eefef6ae6074fd: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:28.399896 11083 tablet_bootstrap.cc:492] T bc8e3edd983e4be9a9726e73f04f08e3 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:28.402191 11065 tablet_bootstrap.cc:654] T 67703d84deb9417bb21bde6365deda2f P 1d34e18a1d394d3f91c9f50d1c7bea4e: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:28.402349 11016 consensus_queue.cc:1035] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:28.403914 11013 consensus_queue.cc:1035] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:28.404047 11061 raft_consensus.cc:2258] T 3371a9d68ee746aca0795e15686ba694 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:28.404923 11061 ts_tablet_manager.cc:1875] T 3371a9d68ee746aca0795e15686ba694 P db0742ee3860468590eefef6ae6074fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 20:10:28.405519 11062 ts_tablet_manager.cc:1909] T 3371a9d68ee746aca0795e15686ba694 P b9957d4086bc4a76b18ed9265a5739d5: Deleting consensus metadata
I20240501 20:10:28.406216 4234 raft_consensus.cc:1261] T 26ed0760e9074bd896e55e99756602bb P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:28.406741 10717 consensus_queue.cc:1035] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:28.409574 7281 raft_consensus.cc:1261] T 26ed0760e9074bd896e55e99756602bb P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:28.411371 10717 consensus_queue.cc:1035] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 20:10:28.413542 7355 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.67:34595, user_credentials={real_user=slave}} blocked reactor thread for 63797.3us
I20240501 20:10:28.418926 11059 tablet_bootstrap.cc:492] T 5f25c23e11dc48ee9d9496c1ae45fcaa P b9957d4086bc4a76b18ed9265a5739d5: No bootstrap required, opened a new log
I20240501 20:10:28.419167 11059 ts_tablet_manager.cc:1367] T 5f25c23e11dc48ee9d9496c1ae45fcaa P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.001s
I20240501 20:10:28.426728 10871 raft_consensus.cc:2932] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:28.431771 7022 raft_consensus.cc:2932] T 797b2374d4214c8eb39982c339a7fbad P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:28.435426 7420 raft_consensus.cc:2932] T 797b2374d4214c8eb39982c339a7fbad P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:28.437815 10717 raft_consensus.cc:2932] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:28.439934 7287 raft_consensus.cc:2932] T 797b2374d4214c8eb39982c339a7fbad P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:28.441869 4088 catalog_manager.cc:5484] T 797b2374d4214c8eb39982c339a7fbad P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: config changed from index 2 to 3, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:28.451085 11057 tablet_bootstrap.cc:654] T 3ffe6b742c3743d8b7de7a0c4105637f P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:28.454917 4233 raft_consensus.cc:2932] T 26ed0760e9074bd896e55e99756602bb P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:28.455480 7152 raft_consensus.cc:1261] T 26ed0760e9074bd896e55e99756602bb P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:28.455827 11080 tablet_bootstrap.cc:492] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:28.456110 11016 raft_consensus.cc:482] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:28.456285 11016 raft_consensus.cc:504] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:28.456985 11016 leader_election.cc:288] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:28.456354 11090 consensus_queue.cc:1035] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:10:28.459599 11058 tablet_bootstrap.cc:492] T 3ffe6b742c3743d8b7de7a0c4105637f P db0742ee3860468590eefef6ae6074fd: No bootstrap required, opened a new log
I20240501 20:10:28.459761 11058 ts_tablet_manager.cc:1367] T 3ffe6b742c3743d8b7de7a0c4105637f P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.148s user 0.003s sys 0.000s
I20240501 20:10:28.459774 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67703d84deb9417bb21bde6365deda2f" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:28.460021 7419 raft_consensus.cc:2454] T 67703d84deb9417bb21bde6365deda2f P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 0.
I20240501 20:10:28.460482 7091 leader_election.cc:302] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:28.460798 11016 raft_consensus.cc:2781] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:28.460986 11016 raft_consensus.cc:482] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:28.461170 11016 raft_consensus.cc:3037] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:28.464797 11016 raft_consensus.cc:504] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:28.465788 11016 leader_election.cc:288] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:28.466702 11058 raft_consensus.cc:348] T 3ffe6b742c3743d8b7de7a0c4105637f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:28.467056 11058 raft_consensus.cc:374] T 3ffe6b742c3743d8b7de7a0c4105637f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:28.467185 11058 raft_consensus.cc:729] T 3ffe6b742c3743d8b7de7a0c4105637f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: FOLLOWER
I20240501 20:10:28.467559 11058 consensus_queue.cc:260] T 3ffe6b742c3743d8b7de7a0c4105637f P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:28.468345 11058 ts_tablet_manager.cc:1398] T 3ffe6b742c3743d8b7de7a0c4105637f P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240501 20:10:28.470340 4088 catalog_manager.cc:5484] T 26ed0760e9074bd896e55e99756602bb P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:28.471796 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67703d84deb9417bb21bde6365deda2f" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:28.472080 7419 raft_consensus.cc:3037] T 67703d84deb9417bb21bde6365deda2f P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:28.457906 4628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67703d84deb9417bb21bde6365deda2f" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:28.470871 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67703d84deb9417bb21bde6365deda2f" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
W20240501 20:10:28.477985 7091 leader_election.cc:341] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Illegal state: must be running to vote when last-logged opid is not known
W20240501 20:10:28.478402 7091 leader_election.cc:341] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:28.479383 11080 tablet_bootstrap.cc:654] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:28.479501 11059 raft_consensus.cc:348] T 5f25c23e11dc48ee9d9496c1ae45fcaa P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:28.479717 11059 raft_consensus.cc:374] T 5f25c23e11dc48ee9d9496c1ae45fcaa P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:28.479827 11059 raft_consensus.cc:729] T 5f25c23e11dc48ee9d9496c1ae45fcaa P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: FOLLOWER
I20240501 20:10:28.480026 11059 consensus_queue.cc:260] T 5f25c23e11dc48ee9d9496c1ae45fcaa P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:28.480345 7154 consensus_queue.cc:237] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:28.480600 11059 ts_tablet_manager.cc:1398] T 5f25c23e11dc48ee9d9496c1ae45fcaa P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.061s user 0.000s sys 0.002s
I20240501 20:10:28.482074 11060 ts_tablet_manager.cc:1888] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:28.482205 11060 log.cc:1196] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/wals/3371a9d68ee746aca0795e15686ba694
I20240501 20:10:28.482501 11061 ts_tablet_manager.cc:1888] T 3371a9d68ee746aca0795e15686ba694 P db0742ee3860468590eefef6ae6074fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 20:10:28.485510 11083 tablet_bootstrap.cc:654] T bc8e3edd983e4be9a9726e73f04f08e3 P 10b47782558c41ee91f5e164a372049a: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:28.491739 11061 log.cc:1196] T 3371a9d68ee746aca0795e15686ba694 P db0742ee3860468590eefef6ae6074fd: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/wals/3371a9d68ee746aca0795e15686ba694
I20240501 20:10:28.492450 7419 raft_consensus.cc:2454] T 67703d84deb9417bb21bde6365deda2f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:28.493060 11061 ts_tablet_manager.cc:1909] T 3371a9d68ee746aca0795e15686ba694 P db0742ee3860468590eefef6ae6074fd: Deleting consensus metadata
I20240501 20:10:28.496059 11062 tablet_replica.cc:330] T d13178f10db0435891b97cebaed6dd5d P b9957d4086bc4a76b18ed9265a5739d5: stopping tablet replica
I20240501 20:10:28.496012 4079 catalog_manager.cc:4830] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): tablet 3371a9d68ee746aca0795e15686ba694 (table rebalancer_dynamic_table_new_name [id=e251cffa507f4bd2adc671356322485b]) successfully deleted
I20240501 20:10:28.496436 11062 raft_consensus.cc:2229] T d13178f10db0435891b97cebaed6dd5d P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:28.496840 11062 raft_consensus.cc:2258] T d13178f10db0435891b97cebaed6dd5d P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:28.500499 11065 tablet_bootstrap.cc:492] T 67703d84deb9417bb21bde6365deda2f P 1d34e18a1d394d3f91c9f50d1c7bea4e: No bootstrap required, opened a new log
I20240501 20:10:28.500782 11065 ts_tablet_manager.cc:1367] T 67703d84deb9417bb21bde6365deda2f P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.115s user 0.001s sys 0.003s
I20240501 20:10:28.494669 7091 leader_election.cc:302] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters: 1d34e18a1d394d3f91c9f50d1c7bea4e
I20240501 20:10:28.501966 11065 raft_consensus.cc:348] T 67703d84deb9417bb21bde6365deda2f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:28.502194 10871 raft_consensus.cc:2781] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:28.502255 11065 raft_consensus.cc:374] T 67703d84deb9417bb21bde6365deda2f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:28.502406 11065 raft_consensus.cc:729] T 67703d84deb9417bb21bde6365deda2f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: FOLLOWER
I20240501 20:10:28.502799 7420 raft_consensus.cc:1261] T 797b2374d4214c8eb39982c339a7fbad P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:28.502811 10871 raft_consensus.cc:686] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Becoming Leader. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Running, Role: LEADER
I20240501 20:10:28.502703 11065 consensus_queue.cc:260] T 67703d84deb9417bb21bde6365deda2f P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:28.499022 7152 raft_consensus.cc:2932] T 26ed0760e9074bd896e55e99756602bb P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:28.503062 10871 consensus_queue.cc:237] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:28.507303 11013 consensus_queue.cc:1035] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:28.498903 7022 raft_consensus.cc:1261] T 797b2374d4214c8eb39982c339a7fbad P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:28.510780 11013 consensus_queue.cc:1035] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:28.514150 11062 ts_tablet_manager.cc:1875] T d13178f10db0435891b97cebaed6dd5d P b9957d4086bc4a76b18ed9265a5739d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:28.515213 11013 raft_consensus.cc:2932] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:28.518214 7420 raft_consensus.cc:2932] T 797b2374d4214c8eb39982c339a7fbad P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:28.521318 4076 catalog_manager.cc:4830] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): tablet 3371a9d68ee746aca0795e15686ba694 (table rebalancer_dynamic_table_new_name [id=e251cffa507f4bd2adc671356322485b]) successfully deleted
I20240501 20:10:28.521764 4076 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 797b2374d4214c8eb39982c339a7fbad with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:28.522459 4365 raft_consensus.cc:1261] T c5b4fbeda51f40b8ac54baf5fb241710 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:28.522459 4498 raft_consensus.cc:1261] T c5b4fbeda51f40b8ac54baf5fb241710 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:28.522881 11099 ts_tablet_manager.cc:902] T ca4f195a6dd14be981395538c2a33ad9 P db0742ee3860468590eefef6ae6074fd: Initiating tablet copy from peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:28.522944 7022 raft_consensus.cc:2932] T 797b2374d4214c8eb39982c339a7fbad P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:28.528692 4627 raft_consensus.cc:1261] T c5b4fbeda51f40b8ac54baf5fb241710 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:28.529780 10980 consensus_queue.cc:1035] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:28.541971 11051 tablet_copy_client.cc:308] T 65b0048212a745c08ab24ead346bf74e P db0742ee3860468590eefef6ae6074fd: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.68:45205
I20240501 20:10:28.544039 4651 tablet_copy_service.cc:140] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Received BeginTabletCopySession request for tablet 65b0048212a745c08ab24ead346bf74e from peer db0742ee3860468590eefef6ae6074fd ({username='slave'} at 127.29.219.70:45393)
I20240501 20:10:28.544255 4651 tablet_copy_service.cc:161] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Beginning new tablet copy session on tablet 65b0048212a745c08ab24ead346bf74e from peer db0742ee3860468590eefef6ae6074fd at {username='slave'} at 127.29.219.70:45393: session id = db0742ee3860468590eefef6ae6074fd-65b0048212a745c08ab24ead346bf74e
I20240501 20:10:28.548341 4651 tablet_copy_source_session.cc:205] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:28.560406 7287 raft_consensus.cc:2932] T 26ed0760e9074bd896e55e99756602bb P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:28.564040 11083 tablet_bootstrap.cc:492] T bc8e3edd983e4be9a9726e73f04f08e3 P 10b47782558c41ee91f5e164a372049a: No bootstrap required, opened a new log
I20240501 20:10:28.564260 11083 ts_tablet_manager.cc:1367] T bc8e3edd983e4be9a9726e73f04f08e3 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.164s user 0.003s sys 0.000s
I20240501 20:10:28.565399 11060 ts_tablet_manager.cc:1909] T 3371a9d68ee746aca0795e15686ba694 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting consensus metadata
I20240501 20:10:28.566547 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet 3371a9d68ee746aca0795e15686ba694 (table rebalancer_dynamic_table_new_name [id=e251cffa507f4bd2adc671356322485b]) successfully deleted
I20240501 20:10:28.567456 11034 raft_consensus.cc:482] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:28.567643 11034 raft_consensus.cc:504] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:28.567873 11057 tablet_bootstrap.cc:492] T 3ffe6b742c3743d8b7de7a0c4105637f P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:28.568022 11057 ts_tablet_manager.cc:1367] T 3ffe6b742c3743d8b7de7a0c4105637f P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.187s user 0.001s sys 0.012s
I20240501 20:10:28.568919 11034 leader_election.cc:288] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:28.568998 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8e3edd983e4be9a9726e73f04f08e3" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
W20240501 20:10:28.569532 4169 leader_election.cc:341] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:10:28.573105 10931 tablet_copy_client.cc:308] T 5215e3f2040747f29aa1c9181a628096 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.70:36321
I20240501 20:10:28.573065 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8e3edd983e4be9a9726e73f04f08e3" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:28.573278 7419 raft_consensus.cc:2454] T bc8e3edd983e4be9a9726e73f04f08e3 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:28.573709 7175 tablet_copy_service.cc:140] P db0742ee3860468590eefef6ae6074fd: Received BeginTabletCopySession request for tablet 5215e3f2040747f29aa1c9181a628096 from peer 3e974c03d3c546f9ab6c91e762a5426b ({username='slave'} at 127.29.219.67:45693)
I20240501 20:10:28.573858 7175 tablet_copy_service.cc:161] P db0742ee3860468590eefef6ae6074fd: Beginning new tablet copy session on tablet 5215e3f2040747f29aa1c9181a628096 from peer 3e974c03d3c546f9ab6c91e762a5426b at {username='slave'} at 127.29.219.67:45693: session id = 3e974c03d3c546f9ab6c91e762a5426b-5215e3f2040747f29aa1c9181a628096
I20240501 20:10:28.574040 4170 leader_election.cc:302] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, dca57823003e4729b769ca12367d7530; no voters: 10b47782558c41ee91f5e164a372049a
I20240501 20:10:28.574828 10958 raft_consensus.cc:2781] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:28.574937 10958 raft_consensus.cc:482] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:28.575028 10958 raft_consensus.cc:3037] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:28.576994 7175 tablet_copy_source_session.cc:205] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:28.577028 11062 ts_tablet_manager.cc:1888] T d13178f10db0435891b97cebaed6dd5d P b9957d4086bc4a76b18ed9265a5739d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:28.577178 11062 log.cc:1196] T d13178f10db0435891b97cebaed6dd5d P b9957d4086bc4a76b18ed9265a5739d5: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/wals/d13178f10db0435891b97cebaed6dd5d
I20240501 20:10:28.568819 11057 raft_consensus.cc:348] T 3ffe6b742c3743d8b7de7a0c4105637f P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:28.577633 11057 raft_consensus.cc:374] T 3ffe6b742c3743d8b7de7a0c4105637f P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:28.577711 11057 raft_consensus.cc:729] T 3ffe6b742c3743d8b7de7a0c4105637f P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:28.578763 11057 consensus_queue.cc:260] T 3ffe6b742c3743d8b7de7a0c4105637f P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:28.558363 4090 catalog_manager.cc:5484] T 797b2374d4214c8eb39982c339a7fbad P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:28.582065 11051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b0048212a745c08ab24ead346bf74e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:28.583544 10700 raft_consensus.cc:1050] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d: attempting to promote NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b to VOTER
I20240501 20:10:28.584429 10700 consensus_queue.cc:237] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:28.586333 4365 consensus_queue.cc:237] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:28.587347 10931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5215e3f2040747f29aa1c9181a628096. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:28.591696 10988 raft_consensus.cc:482] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:28.591884 10988 raft_consensus.cc:504] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:28.592499 10988 leader_election.cc:288] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:28.594521 7152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ffe6b742c3743d8b7de7a0c4105637f" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:28.594806 7152 raft_consensus.cc:2454] T 3ffe6b742c3743d8b7de7a0c4105637f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:10:28.595350 4562 leader_election.cc:302] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:28.595969 11077 raft_consensus.cc:482] T bc8e3edd983e4be9a9726e73f04f08e3 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:28.596244 10988 raft_consensus.cc:2781] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:28.596323 10988 raft_consensus.cc:482] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:28.596377 10988 raft_consensus.cc:3037] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:28.596175 11077 raft_consensus.cc:504] T bc8e3edd983e4be9a9726e73f04f08e3 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:28.597019 11077 leader_election.cc:288] T bc8e3edd983e4be9a9726e73f04f08e3 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:28.570024 7417 raft_consensus.cc:2932] T 2aa1a5f266e840698f2127289628dcba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:28.599891 11083 raft_consensus.cc:348] T bc8e3edd983e4be9a9726e73f04f08e3 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:28.600162 11083 raft_consensus.cc:374] T bc8e3edd983e4be9a9726e73f04f08e3 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:28.600247 11083 raft_consensus.cc:729] T bc8e3edd983e4be9a9726e73f04f08e3 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: FOLLOWER
I20240501 20:10:28.600508 11083 consensus_queue.cc:260] T bc8e3edd983e4be9a9726e73f04f08e3 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:28.601112 11083 ts_tablet_manager.cc:1398] T bc8e3edd983e4be9a9726e73f04f08e3 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.037s user 0.001s sys 0.000s
I20240501 20:10:28.604650 7281 raft_consensus.cc:1261] T 26ed0760e9074bd896e55e99756602bb P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:28.604687 10988 raft_consensus.cc:504] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:28.605437 10885 consensus_queue.cc:1035] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:28.605685 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ffe6b742c3743d8b7de7a0c4105637f" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:28.605895 7153 raft_consensus.cc:3037] T 3ffe6b742c3743d8b7de7a0c4105637f P db0742ee3860468590eefef6ae6074fd [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:28.608292 7267 tablet_service.cc:1514] Processing DeleteTablet for tablet 797b2374d4214c8eb39982c339a7fbad with delete_type TABLET_DATA_TOMBSTONED (TS bceeae1c79964acabdae8ff771bf0c97 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54166
I20240501 20:10:28.608693 7153 raft_consensus.cc:2454] T 3ffe6b742c3743d8b7de7a0c4105637f P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:10:28.609390 4562 leader_election.cc:302] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:28.609687 10377 raft_consensus.cc:2781] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:28.609952 7281 raft_consensus.cc:2932] T 26ed0760e9074bd896e55e99756602bb P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:28.610800 10988 leader_election.cc:288] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 election: Requested vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:28.611882 11094 raft_consensus.cc:2932] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:28.579221 10958 raft_consensus.cc:504] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:28.614753 10958 leader_election.cc:288] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:28.593468 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ffe6b742c3743d8b7de7a0c4105637f" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:28.604156 4233 raft_consensus.cc:1261] T 26ed0760e9074bd896e55e99756602bb P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:28.610352 4232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ffe6b742c3743d8b7de7a0c4105637f" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:10:28.617406 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8e3edd983e4be9a9726e73f04f08e3" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:10:28.617592 4365 raft_consensus.cc:3037] T bc8e3edd983e4be9a9726e73f04f08e3 P 10b47782558c41ee91f5e164a372049a [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:28.620690 4365 raft_consensus.cc:2454] T bc8e3edd983e4be9a9726e73f04f08e3 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:28.621167 4169 leader_election.cc:302] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:28.621913 10958 raft_consensus.cc:2781] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:28.622072 10958 raft_consensus.cc:686] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:10:28.622223 4234 raft_consensus.cc:2454] T 3ffe6b742c3743d8b7de7a0c4105637f P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 0.
I20240501 20:10:28.622628 10958 consensus_queue.cc:237] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:28.624540 10988 raft_consensus.cc:482] T 67703d84deb9417bb21bde6365deda2f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:28.624676 10988 raft_consensus.cc:504] T 67703d84deb9417bb21bde6365deda2f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:28.621856 10717 consensus_queue.cc:1035] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:28.625782 7152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67703d84deb9417bb21bde6365deda2f" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:28.626022 10988 leader_election.cc:288] T 67703d84deb9417bb21bde6365deda2f P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:28.627032 4234 raft_consensus.cc:2932] T 26ed0760e9074bd896e55e99756602bb P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
W20240501 20:10:28.629966 4302 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.72:39817, user_credentials={real_user=slave}} blocked reactor thread for 86416.9us
I20240501 20:10:28.631610 4088 catalog_manager.cc:5484] T 26ed0760e9074bd896e55e99756602bb P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:28.640986 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8e3edd983e4be9a9726e73f04f08e3" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:28.642452 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8e3edd983e4be9a9726e73f04f08e3" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:28.642688 7419 raft_consensus.cc:3037] T bc8e3edd983e4be9a9726e73f04f08e3 P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:28.642093 4365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc8e3edd983e4be9a9726e73f04f08e3" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:10:28.645892 4365 raft_consensus.cc:2379] T bc8e3edd983e4be9a9726e73f04f08e3 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9957d4086bc4a76b18ed9265a5739d5 in current term 1: Already voted for candidate dca57823003e4729b769ca12367d7530 in this term.
I20240501 20:10:28.647904 11065 ts_tablet_manager.cc:1398] T 67703d84deb9417bb21bde6365deda2f P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.147s user 0.002s sys 0.000s
I20240501 20:10:28.646761 11101 consensus_queue.cc:1035] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20240501 20:10:28.652503 7419 raft_consensus.cc:2454] T bc8e3edd983e4be9a9726e73f04f08e3 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:28.652794 11078 consensus_queue.cc:1035] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:28.653635 4089 catalog_manager.cc:5484] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd reported cstate change: term changed from 0 to 1, leader changed from <none> to db0742ee3860468590eefef6ae6074fd (127.29.219.70). New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:28.655313 11057 ts_tablet_manager.cc:1398] T 3ffe6b742c3743d8b7de7a0c4105637f P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.087s user 0.002s sys 0.000s
I20240501 20:10:28.656899 7416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67703d84deb9417bb21bde6365deda2f" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:28.657155 7416 raft_consensus.cc:2379] T 67703d84deb9417bb21bde6365deda2f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in current term 1: Already voted for candidate db0742ee3860468590eefef6ae6074fd in this term.
I20240501 20:10:28.657451 7355 leader_election.cc:302] T bc8e3edd983e4be9a9726e73f04f08e3 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5; no voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530
I20240501 20:10:28.657775 11078 raft_consensus.cc:2726] T bc8e3edd983e4be9a9726e73f04f08e3 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:28.658967 11057 tablet_bootstrap.cc:492] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:28.668929 10377 raft_consensus.cc:686] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Becoming Leader. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Running, Role: LEADER
I20240501 20:10:28.669381 11117 tablet_replica.cc:330] T 797b2374d4214c8eb39982c339a7fbad P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:28.669976 11117 raft_consensus.cc:2229] T 797b2374d4214c8eb39982c339a7fbad P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:28.669996 10377 consensus_queue.cc:237] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:28.670305 11117 raft_consensus.cc:2258] T 797b2374d4214c8eb39982c339a7fbad P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:28.671540 11117 ts_tablet_manager.cc:1875] T 797b2374d4214c8eb39982c339a7fbad P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:28.671998 4079 catalog_manager.cc:4830] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): tablet d13178f10db0435891b97cebaed6dd5d (table rebalancer_extra_table_10 [id=abd146b01560429fae00683c20d75e9c]) successfully deleted
I20240501 20:10:28.673064 7135 tablet_service.cc:1514] Processing DeleteTablet for tablet 26ed0760e9074bd896e55e99756602bb with delete_type TABLET_DATA_TOMBSTONED (TS db0742ee3860468590eefef6ae6074fd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52036
I20240501 20:10:28.673476 11061 tablet_replica.cc:330] T 26ed0760e9074bd896e55e99756602bb P db0742ee3860468590eefef6ae6074fd: stopping tablet replica
I20240501 20:10:28.673782 11061 raft_consensus.cc:2229] T 26ed0760e9074bd896e55e99756602bb P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:28.674036 11061 raft_consensus.cc:2258] T 26ed0760e9074bd896e55e99756602bb P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:28.675109 11061 ts_tablet_manager.cc:1875] T 26ed0760e9074bd896e55e99756602bb P db0742ee3860468590eefef6ae6074fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:28.675542 4087 catalog_manager.cc:5484] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:28.688467 10931 tablet_copy_client.cc:783] T 5215e3f2040747f29aa1c9181a628096 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:28.689112 10931 tablet_copy_client.cc:647] T 5215e3f2040747f29aa1c9181a628096 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:28.689195 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 26ed0760e9074bd896e55e99756602bb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:28.691798 11061 ts_tablet_manager.cc:1888] T 26ed0760e9074bd896e55e99756602bb P db0742ee3860468590eefef6ae6074fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:28.692080 11061 log.cc:1196] T 26ed0760e9074bd896e55e99756602bb P db0742ee3860468590eefef6ae6074fd: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/wals/26ed0760e9074bd896e55e99756602bb
I20240501 20:10:28.692857 4565 leader_election.cc:302] T 67703d84deb9417bb21bde6365deda2f P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e; no voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd
I20240501 20:10:28.693348 11118 raft_consensus.cc:3037] T 67703d84deb9417bb21bde6365deda2f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:28.693518 11057 tablet_bootstrap.cc:654] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:28.696444 11034 raft_consensus.cc:1050] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530: attempting to promote NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e to VOTER
I20240501 20:10:28.697474 11034 consensus_queue.cc:237] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } }
I20240501 20:10:28.702270 4076 catalog_manager.cc:4830] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): tablet 26ed0760e9074bd896e55e99756602bb (table rebalancer_extra_table_26 [id=306aca4fdfa94f4f81f64f300d602707]) successfully deleted
I20240501 20:10:28.705808 10931 tablet_copy_client.cc:523] T 5215e3f2040747f29aa1c9181a628096 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:28.707381 11118 raft_consensus.cc:2726] T 67703d84deb9417bb21bde6365deda2f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:10:28.708410 4087 catalog_manager.cc:5484] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68). New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:28.711416 11051 tablet_copy_client.cc:783] T 65b0048212a745c08ab24ead346bf74e P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:28.715216 11051 tablet_copy_client.cc:647] T 65b0048212a745c08ab24ead346bf74e P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:28.718127 11080 tablet_bootstrap.cc:492] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97: No bootstrap required, opened a new log
I20240501 20:10:28.720312 11080 ts_tablet_manager.cc:1367] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.265s user 0.014s sys 0.000s
I20240501 20:10:28.721478 11051 tablet_copy_client.cc:523] T 65b0048212a745c08ab24ead346bf74e P db0742ee3860468590eefef6ae6074fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:28.723567 11080 raft_consensus.cc:348] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:28.724298 11080 raft_consensus.cc:374] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:28.724565 11080 raft_consensus.cc:729] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: FOLLOWER
I20240501 20:10:28.724716 11101 raft_consensus.cc:2932] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:28.725548 11080 consensus_queue.cc:260] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:28.728361 11080 ts_tablet_manager.cc:1398] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 20:10:28.729959 4628 raft_consensus.cc:2932] T c5b4fbeda51f40b8ac54baf5fb241710 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:28.739912 4084 catalog_manager.cc:5484] T c5b4fbeda51f40b8ac54baf5fb241710 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:28.741142 11051 tablet_bootstrap.cc:492] T 65b0048212a745c08ab24ead346bf74e P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:28.742216 4496 raft_consensus.cc:2932] T c5b4fbeda51f40b8ac54baf5fb241710 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:28.743232 11130 ts_tablet_manager.cc:902] T 2aa1a5f266e840698f2127289628dcba P ef0a4ada548446b3a531cb39a3c4e38d: Initiating tablet copy from peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:28.748225 11117 ts_tablet_manager.cc:1888] T 797b2374d4214c8eb39982c339a7fbad P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:28.749513 10931 tablet_bootstrap.cc:492] T 5215e3f2040747f29aa1c9181a628096 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:28.752635 11099 tablet_copy_client.cc:308] T ca4f195a6dd14be981395538c2a33ad9 P db0742ee3860468590eefef6ae6074fd: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.67:34595
I20240501 20:10:28.753248 4519 tablet_copy_service.cc:140] P 3e974c03d3c546f9ab6c91e762a5426b: Received BeginTabletCopySession request for tablet ca4f195a6dd14be981395538c2a33ad9 from peer db0742ee3860468590eefef6ae6074fd ({username='slave'} at 127.29.219.70:56941)
I20240501 20:10:28.753367 4519 tablet_copy_service.cc:161] P 3e974c03d3c546f9ab6c91e762a5426b: Beginning new tablet copy session on tablet ca4f195a6dd14be981395538c2a33ad9 from peer db0742ee3860468590eefef6ae6074fd at {username='slave'} at 127.29.219.70:56941: session id = db0742ee3860468590eefef6ae6074fd-ca4f195a6dd14be981395538c2a33ad9
I20240501 20:10:28.756260 4519 tablet_copy_source_session.cc:205] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:28.757691 11099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4f195a6dd14be981395538c2a33ad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:28.761075 11117 log.cc:1196] T 797b2374d4214c8eb39982c339a7fbad P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/797b2374d4214c8eb39982c339a7fbad
I20240501 20:10:28.762683 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet 797b2374d4214c8eb39982c339a7fbad (table rebalancer_extra_table_26 [id=306aca4fdfa94f4f81f64f300d602707]) successfully deleted
I20240501 20:10:28.763594 4363 raft_consensus.cc:2932] T c5b4fbeda51f40b8ac54baf5fb241710 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:28.766225 11057 tablet_bootstrap.cc:492] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530: No bootstrap required, opened a new log
I20240501 20:10:28.767005 11057 ts_tablet_manager.cc:1367] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.108s user 0.002s sys 0.005s
I20240501 20:10:28.767907 7420 consensus_queue.cc:237] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } }
I20240501 20:10:28.769922 7416 raft_consensus.cc:1261] T 48528ec242974de4ac20eb0161f0cb05 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:28.768193 11057 raft_consensus.cc:348] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:28.770603 11057 raft_consensus.cc:374] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:28.772205 4499 raft_consensus.cc:1261] T c5b4fbeda51f40b8ac54baf5fb241710 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:28.772229 11057 raft_consensus.cc:729] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: FOLLOWER
I20240501 20:10:28.772605 11057 consensus_queue.cc:260] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:28.772656 10701 consensus_queue.cc:1035] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:28.773572 11078 consensus_queue.cc:1035] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:28.773864 11057 ts_tablet_manager.cc:1398] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 20:10:28.778201 4496 raft_consensus.cc:1261] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:28.778980 10701 consensus_queue.cc:1035] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:28.786306 11099 tablet_copy_client.cc:783] T ca4f195a6dd14be981395538c2a33ad9 P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:28.786726 7152 raft_consensus.cc:1261] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:28.787914 11099 tablet_copy_client.cc:647] T ca4f195a6dd14be981395538c2a33ad9 P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:28.788203 10701 consensus_queue.cc:1035] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:28.788993 10958 raft_consensus.cc:482] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:28.789222 10958 raft_consensus.cc:504] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:28.791236 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f25c23e11dc48ee9d9496c1ae45fcaa" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:28.791496 7287 raft_consensus.cc:2454] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 0.
I20240501 20:10:28.791997 4168 leader_election.cc:302] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:28.792337 10672 raft_consensus.cc:2781] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:28.792426 10672 raft_consensus.cc:482] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:28.792534 10672 raft_consensus.cc:3037] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:28.796059 4628 raft_consensus.cc:1261] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:28.796479 10672 raft_consensus.cc:504] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:28.796751 10350 consensus_queue.cc:1035] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:28.797928 11099 tablet_copy_client.cc:523] T ca4f195a6dd14be981395538c2a33ad9 P db0742ee3860468590eefef6ae6074fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:28.797930 7287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f25c23e11dc48ee9d9496c1ae45fcaa" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:28.798166 7287 raft_consensus.cc:3037] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:28.801505 7287 raft_consensus.cc:2454] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:28.802264 4168 leader_election.cc:302] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:28.802714 10350 raft_consensus.cc:2781] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:28.804445 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f25c23e11dc48ee9d9496c1ae45fcaa" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:28.804690 7419 raft_consensus.cc:3037] T 5f25c23e11dc48ee9d9496c1ae45fcaa P b9957d4086bc4a76b18ed9265a5739d5 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:28.808984 7419 raft_consensus.cc:2454] T 5f25c23e11dc48ee9d9496c1ae45fcaa P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:28.809895 10672 leader_election.cc:288] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 election: Requested vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:28.811509 11099 tablet_bootstrap.cc:492] T ca4f195a6dd14be981395538c2a33ad9 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:28.816437 10958 leader_election.cc:288] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:28.820730 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f25c23e11dc48ee9d9496c1ae45fcaa" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:28.822126 7420 raft_consensus.cc:2362] T 5f25c23e11dc48ee9d9496c1ae45fcaa P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1. Re-sending same reply.
I20240501 20:10:28.829799 11078 raft_consensus.cc:2932] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:28.835458 4079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c5b4fbeda51f40b8ac54baf5fb241710 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:28.840481 10350 raft_consensus.cc:686] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
I20240501 20:10:28.840937 10350 consensus_queue.cc:237] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:28.844400 4499 raft_consensus.cc:2932] T c5b4fbeda51f40b8ac54baf5fb241710 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:28.850085 4088 catalog_manager.cc:5484] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: config changed from index 3 to 4, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:28.860438 4089 catalog_manager.cc:5484] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 0 to 1, leader changed from <none> to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } }
W20240501 20:10:28.862586 4169 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.67:34595, user_credentials={real_user=slave}} blocked reactor thread for 53381us
I20240501 20:10:28.863426 4499 raft_consensus.cc:1261] T 00ae88161b514e4aa4b8d134a19de001 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:28.863341 7154 consensus_queue.cc:237] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:28.864360 10701 raft_consensus.cc:2932] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:28.867374 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet c5b4fbeda51f40b8ac54baf5fb241710 with delete_type TABLET_DATA_TOMBSTONED (TS 1d34e18a1d394d3f91c9f50d1c7bea4e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:28.871012 4499 raft_consensus.cc:2932] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:28.871870 7152 raft_consensus.cc:2932] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:28.883639 10672 consensus_queue.cc:1035] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:28.885697 7022 raft_consensus.cc:1261] T 00ae88161b514e4aa4b8d134a19de001 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:28.889411 11149 tablet_replica.cc:330] T c5b4fbeda51f40b8ac54baf5fb241710 P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:28.888204 7416 raft_consensus.cc:2932] T 48528ec242974de4ac20eb0161f0cb05 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:28.896636 11149 raft_consensus.cc:2229] T c5b4fbeda51f40b8ac54baf5fb241710 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:28.896958 11149 raft_consensus.cc:2258] T c5b4fbeda51f40b8ac54baf5fb241710 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:28.896968 4088 catalog_manager.cc:5484] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:28.900363 7287 raft_consensus.cc:1261] T b2619f3bed304d3ab864f56d5b655574 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:28.901800 7090 consensus_peers.cc:485] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2619f3bed304d3ab864f56d5b655574. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:28.901983 10871 consensus_queue.cc:1035] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:28.904824 7022 consensus_queue.cc:237] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:28.907411 7153 raft_consensus.cc:1261] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:28.908720 11111 consensus_queue.cc:1035] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:28.910919 4627 raft_consensus.cc:1261] T b2619f3bed304d3ab864f56d5b655574 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:28.912809 11016 consensus_queue.cc:1035] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:28.914958 4496 raft_consensus.cc:1261] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:28.917001 10700 consensus_queue.cc:1035] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:28.917104 7154 raft_consensus.cc:2932] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:28.920137 10672 consensus_queue.cc:1035] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:28.925220 11149 ts_tablet_manager.cc:1875] T c5b4fbeda51f40b8ac54baf5fb241710 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:28.929949 10931 tablet_bootstrap.cc:492] T 5215e3f2040747f29aa1c9181a628096 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:28.930390 11016 consensus_queue.cc:1035] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:28.934643 10931 tablet_bootstrap.cc:492] T 5215e3f2040747f29aa1c9181a628096 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap complete.
I20240501 20:10:28.935356 10931 ts_tablet_manager.cc:1367] T 5215e3f2040747f29aa1c9181a628096 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.186s user 0.005s sys 0.009s
I20240501 20:10:28.929540 4499 raft_consensus.cc:2932] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:28.945086 10672 raft_consensus.cc:2932] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:28.945555 11013 raft_consensus.cc:2932] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:28.947355 4627 raft_consensus.cc:2932] T b2619f3bed304d3ab864f56d5b655574 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:28.950313 4628 raft_consensus.cc:2932] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:28.950978 7022 raft_consensus.cc:2932] T 00ae88161b514e4aa4b8d134a19de001 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:28.954509 7281 raft_consensus.cc:2932] T b2619f3bed304d3ab864f56d5b655574 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:28.954762 4088 catalog_manager.cc:5484] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:28.957795 4089 catalog_manager.cc:5484] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:28.967124 4084 catalog_manager.cc:5484] T b2619f3bed304d3ab864f56d5b655574 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:28.937376 10931 raft_consensus.cc:348] T 5215e3f2040747f29aa1c9181a628096 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:28.968708 10931 raft_consensus.cc:729] T 5215e3f2040747f29aa1c9181a628096 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: LEARNER
I20240501 20:10:28.968935 10931 consensus_queue.cc:260] T 5215e3f2040747f29aa1c9181a628096 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:28.971015 4365 raft_consensus.cc:1261] T c5b4fbeda51f40b8ac54baf5fb241710 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:28.977313 4234 consensus_queue.cc:237] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } }
I20240501 20:10:28.980440 11077 consensus_queue.cc:1035] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:28.980605 4496 raft_consensus.cc:1261] T 00ae88161b514e4aa4b8d134a19de001 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:28.981215 11144 consensus_queue.cc:1035] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:28.983072 11130 tablet_copy_client.cc:308] T 2aa1a5f266e840698f2127289628dcba P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.67:34595
I20240501 20:10:28.984098 4519 tablet_copy_service.cc:140] P 3e974c03d3c546f9ab6c91e762a5426b: Received BeginTabletCopySession request for tablet 2aa1a5f266e840698f2127289628dcba from peer ef0a4ada548446b3a531cb39a3c4e38d ({username='slave'} at 127.29.219.69:60767)
I20240501 20:10:28.984225 4519 tablet_copy_service.cc:161] P 3e974c03d3c546f9ab6c91e762a5426b: Beginning new tablet copy session on tablet 2aa1a5f266e840698f2127289628dcba from peer ef0a4ada548446b3a531cb39a3c4e38d at {username='slave'} at 127.29.219.69:60767: session id = ef0a4ada548446b3a531cb39a3c4e38d-2aa1a5f266e840698f2127289628dcba
I20240501 20:10:28.984795 4363 raft_consensus.cc:2932] T c5b4fbeda51f40b8ac54baf5fb241710 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:28.985188 4628 raft_consensus.cc:1261] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:28.985723 11144 consensus_queue.cc:1035] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:28.988073 7400 tablet_service.cc:1514] Processing DeleteTablet for tablet 48528ec242974de4ac20eb0161f0cb05 with delete_type TABLET_DATA_TOMBSTONED (TS b9957d4086bc4a76b18ed9265a5739d5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40182
I20240501 20:10:28.991264 4631 raft_consensus.cc:2932] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:28.991147 11034 raft_consensus.cc:2932] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:28.982381 4496 raft_consensus.cc:2932] T 00ae88161b514e4aa4b8d134a19de001 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:28.999857 11013 consensus_queue.cc:1035] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:29.000474 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 00ae88161b514e4aa4b8d134a19de001 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:29.001222 10931 ts_tablet_manager.cc:1398] T 5215e3f2040747f29aa1c9181a628096 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.065s user 0.002s sys 0.000s
I20240501 20:10:29.004530 4519 tablet_copy_source_session.cc:205] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:29.006120 7175 tablet_copy_service.cc:342] P db0742ee3860468590eefef6ae6074fd: Request end of tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-5215e3f2040747f29aa1c9181a628096 received from {username='slave'} at 127.29.219.67:45693
I20240501 20:10:29.006263 7175 tablet_copy_service.cc:433] P db0742ee3860468590eefef6ae6074fd: ending tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-5215e3f2040747f29aa1c9181a628096 on tablet 5215e3f2040747f29aa1c9181a628096 with peer 3e974c03d3c546f9ab6c91e762a5426b
I20240501 20:10:29.007205 4087 catalog_manager.cc:5484] T 00ae88161b514e4aa4b8d134a19de001 P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:29.012025 4498 raft_consensus.cc:2932] T 00ae88161b514e4aa4b8d134a19de001 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:29.013455 11149 ts_tablet_manager.cc:1888] T c5b4fbeda51f40b8ac54baf5fb241710 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:29.013573 11149 log.cc:1196] T c5b4fbeda51f40b8ac54baf5fb241710 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/c5b4fbeda51f40b8ac54baf5fb241710
I20240501 20:10:29.014425 11130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aa1a5f266e840698f2127289628dcba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:29.030905 11062 tablet_replica.cc:330] T 48528ec242974de4ac20eb0161f0cb05 P b9957d4086bc4a76b18ed9265a5739d5: stopping tablet replica
I20240501 20:10:29.030926 11146 consensus_queue.cc:1035] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:29.031167 11062 raft_consensus.cc:2229] T 48528ec242974de4ac20eb0161f0cb05 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:29.031384 11062 raft_consensus.cc:2258] T 48528ec242974de4ac20eb0161f0cb05 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:29.032585 11062 ts_tablet_manager.cc:1875] T 48528ec242974de4ac20eb0161f0cb05 P b9957d4086bc4a76b18ed9265a5739d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:29.028847 11111 raft_consensus.cc:2932] T 48528ec242974de4ac20eb0161f0cb05 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:29.040788 11062 ts_tablet_manager.cc:1888] T 48528ec242974de4ac20eb0161f0cb05 P b9957d4086bc4a76b18ed9265a5739d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:29.041159 11062 log.cc:1196] T 48528ec242974de4ac20eb0161f0cb05 P b9957d4086bc4a76b18ed9265a5739d5: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/wals/48528ec242974de4ac20eb0161f0cb05
I20240501 20:10:29.026209 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet c5b4fbeda51f40b8ac54baf5fb241710 (table rebalancer_extra_table_32 [id=338f819ddccb42a3ae45044c701a6eb5]) successfully deleted
I20240501 20:10:29.049896 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 48528ec242974de4ac20eb0161f0cb05 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:29.050331 4079 catalog_manager.cc:4830] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): tablet 48528ec242974de4ac20eb0161f0cb05 (table rebalancer_extra_table_20 [id=880fdf37926743948f9ffe03bdaeeaa5]) successfully deleted
I20240501 20:10:29.052980 4093 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44422:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 20:10:29.067693 7001 tablet_service.cc:1514] Processing DeleteTablet for tablet 00ae88161b514e4aa4b8d134a19de001 with delete_type TABLET_DATA_TOMBSTONED (TS ef0a4ada548446b3a531cb39a3c4e38d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:29.068087 11060 tablet_replica.cc:330] T 00ae88161b514e4aa4b8d134a19de001 P ef0a4ada548446b3a531cb39a3c4e38d: stopping tablet replica
I20240501 20:10:29.068324 11060 raft_consensus.cc:2229] T 00ae88161b514e4aa4b8d134a19de001 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:29.068583 11060 raft_consensus.cc:2258] T 00ae88161b514e4aa4b8d134a19de001 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:29.069782 11060 ts_tablet_manager.cc:1875] T 00ae88161b514e4aa4b8d134a19de001 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:29.081061 11060 ts_tablet_manager.cc:1888] T 00ae88161b514e4aa4b8d134a19de001 P ef0a4ada548446b3a531cb39a3c4e38d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:29.084777 11060 log.cc:1196] T 00ae88161b514e4aa4b8d134a19de001 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/wals/00ae88161b514e4aa4b8d134a19de001
I20240501 20:10:29.089298 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet 00ae88161b514e4aa4b8d134a19de001 (table rebalancer_extra_table_34 [id=513576dc865743a18e99f37632b2a4a2]) successfully deleted
I20240501 20:10:29.089963 11130 tablet_copy_client.cc:783] T 2aa1a5f266e840698f2127289628dcba P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:29.103121 11130 tablet_copy_client.cc:647] T 2aa1a5f266e840698f2127289628dcba P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:29.117914 11130 tablet_copy_client.cc:523] T 2aa1a5f266e840698f2127289628dcba P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:29.123348 11130 tablet_bootstrap.cc:492] T 2aa1a5f266e840698f2127289628dcba P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:29.140735 7287 raft_consensus.cc:1261] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:29.141259 11034 consensus_queue.cc:1035] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:29.154556 7416 raft_consensus.cc:1261] T 5f25c23e11dc48ee9d9496c1ae45fcaa P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:29.160923 4084 catalog_manager.cc:5576] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941) does not have the latest schema for tablet 5f25c23e11dc48ee9d9496c1ae45fcaa (table rebalancer_dynamic_table [id=ec12d35b5fd7479a8edeb64964c33483]). Expected version 1 got 0
I20240501 20:10:29.168253 11143 consensus_queue.cc:1035] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:29.168742 10350 consensus_queue.cc:1035] T bc8e3edd983e4be9a9726e73f04f08e3 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:29.171242 4234 raft_consensus.cc:3037] T 3ffe6b742c3743d8b7de7a0c4105637f P dca57823003e4729b769ca12367d7530 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:29.172700 4499 raft_consensus.cc:1203] T 5215e3f2040747f29aa1c9181a628096 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 20:10:29.179702 11118 consensus_queue.cc:1035] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:29.189334 4089 catalog_manager.cc:5576] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817) does not have the latest schema for tablet 5f25c23e11dc48ee9d9496c1ae45fcaa (table rebalancer_dynamic_table [id=ec12d35b5fd7479a8edeb64964c33483]). Expected version 1 got 0
I20240501 20:10:29.194857 7022 consensus_queue.cc:237] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:29.209568 11178 tablet.cc:1722] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:29.213263 10377 consensus_queue.cc:1035] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:29.216845 11178 tablet.cc:1620] T 5f25c23e11dc48ee9d9496c1ae45fcaa P dca57823003e4729b769ca12367d7530: MemRowSet was empty: no flush needed.
I20240501 20:10:29.217654 4499 raft_consensus.cc:1261] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:29.218518 6957 consensus_peers.cc:485] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d860bb204134f0994462275519feecf. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:29.218890 11137 consensus_queue.cc:1035] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:29.234237 4088 catalog_manager.cc:6029] T 00000000000000000000000000000000 P ee64e9b209b3427cb57a00ccc1cd5617: rebalancer_dynamic_table [id=ec12d35b5fd7479a8edeb64964c33483] alter complete (version 1)
I20240501 20:10:29.247596 4087 catalog_manager.cc:5576] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447) does not have the latest schema for tablet 5f25c23e11dc48ee9d9496c1ae45fcaa (table rebalancer_dynamic_table [id=ec12d35b5fd7479a8edeb64964c33483]). Expected version 1 got 0
I20240501 20:10:29.254588 7153 raft_consensus.cc:1261] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:29.256604 11184 tablet.cc:1722] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:29.257273 11184 tablet.cc:1620] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97: MemRowSet was empty: no flush needed.
I20240501 20:10:29.263497 11111 consensus_queue.cc:1035] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20240501 20:10:29.265781 4088 catalog_manager.cc:5576] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447) does not have the latest schema for tablet 5f25c23e11dc48ee9d9496c1ae45fcaa (table rebalancer_dynamic_table [id=ec12d35b5fd7479a8edeb64964c33483]). Expected version 1 got 0
I20240501 20:10:29.278405 11179 tablet.cc:1722] T 5f25c23e11dc48ee9d9496c1ae45fcaa P b9957d4086bc4a76b18ed9265a5739d5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 20:10:29.288096 4089 catalog_manager.cc:5576] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817) does not have the latest schema for tablet 5f25c23e11dc48ee9d9496c1ae45fcaa (table rebalancer_dynamic_table [id=ec12d35b5fd7479a8edeb64964c33483]). Expected version 1 got 0
I20240501 20:10:29.293355 11179 tablet.cc:1620] T 5f25c23e11dc48ee9d9496c1ae45fcaa P b9957d4086bc4a76b18ed9265a5739d5: MemRowSet was empty: no flush needed.
I20240501 20:10:29.293455 11111 raft_consensus.cc:2932] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:29.298806 4498 raft_consensus.cc:2932] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:29.312846 4089 catalog_manager.cc:5484] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:29.319718 7152 raft_consensus.cc:2932] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:29.332283 11130 tablet_bootstrap.cc:492] T 2aa1a5f266e840698f2127289628dcba P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:29.332821 11130 tablet_bootstrap.cc:492] T 2aa1a5f266e840698f2127289628dcba P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap complete.
I20240501 20:10:29.333130 11130 ts_tablet_manager.cc:1367] T 2aa1a5f266e840698f2127289628dcba P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.210s user 0.059s sys 0.003s
I20240501 20:10:29.333964 11130 raft_consensus.cc:348] T 2aa1a5f266e840698f2127289628dcba P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:29.334280 11130 raft_consensus.cc:729] T 2aa1a5f266e840698f2127289628dcba P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: LEARNER
I20240501 20:10:29.334496 11130 consensus_queue.cc:260] T 2aa1a5f266e840698f2127289628dcba P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:29.343190 11130 ts_tablet_manager.cc:1398] T 2aa1a5f266e840698f2127289628dcba P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240501 20:10:29.343917 4519 tablet_copy_service.cc:342] P 3e974c03d3c546f9ab6c91e762a5426b: Request end of tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-2aa1a5f266e840698f2127289628dcba received from {username='slave'} at 127.29.219.69:60767
I20240501 20:10:29.344030 4519 tablet_copy_service.cc:433] P 3e974c03d3c546f9ab6c91e762a5426b: ending tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-2aa1a5f266e840698f2127289628dcba on tablet 2aa1a5f266e840698f2127289628dcba with peer ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:29.406224 11051 tablet_bootstrap.cc:492] T 65b0048212a745c08ab24ead346bf74e P db0742ee3860468590eefef6ae6074fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:29.421062 11051 tablet_bootstrap.cc:492] T 65b0048212a745c08ab24ead346bf74e P db0742ee3860468590eefef6ae6074fd: Bootstrap complete.
I20240501 20:10:29.421447 11051 ts_tablet_manager.cc:1367] T 65b0048212a745c08ab24ead346bf74e P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.680s user 0.048s sys 0.013s
I20240501 20:10:29.422466 11051 raft_consensus.cc:348] T 65b0048212a745c08ab24ead346bf74e P db0742ee3860468590eefef6ae6074fd [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:29.422834 11051 raft_consensus.cc:729] T 65b0048212a745c08ab24ead346bf74e P db0742ee3860468590eefef6ae6074fd [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: LEARNER
I20240501 20:10:29.423102 11051 consensus_queue.cc:260] T 65b0048212a745c08ab24ead346bf74e P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:29.423821 11051 ts_tablet_manager.cc:1398] T 65b0048212a745c08ab24ead346bf74e P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:29.424553 4651 tablet_copy_service.cc:342] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Request end of tablet copy session db0742ee3860468590eefef6ae6074fd-65b0048212a745c08ab24ead346bf74e received from {username='slave'} at 127.29.219.70:45393
I20240501 20:10:29.424666 4651 tablet_copy_service.cc:433] P 1d34e18a1d394d3f91c9f50d1c7bea4e: ending tablet copy session db0742ee3860468590eefef6ae6074fd-65b0048212a745c08ab24ead346bf74e on tablet 65b0048212a745c08ab24ead346bf74e with peer db0742ee3860468590eefef6ae6074fd
I20240501 20:10:29.476244 10931 ts_tablet_manager.cc:902] T b2619f3bed304d3ab864f56d5b655574 P 3e974c03d3c546f9ab6c91e762a5426b: Initiating tablet copy from peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:29.511561 10931 tablet_copy_client.cc:308] T b2619f3bed304d3ab864f56d5b655574 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.70:36321
I20240501 20:10:29.512290 7175 tablet_copy_service.cc:140] P db0742ee3860468590eefef6ae6074fd: Received BeginTabletCopySession request for tablet b2619f3bed304d3ab864f56d5b655574 from peer 3e974c03d3c546f9ab6c91e762a5426b ({username='slave'} at 127.29.219.67:45693)
I20240501 20:10:29.512527 7175 tablet_copy_service.cc:161] P db0742ee3860468590eefef6ae6074fd: Beginning new tablet copy session on tablet b2619f3bed304d3ab864f56d5b655574 from peer 3e974c03d3c546f9ab6c91e762a5426b at {username='slave'} at 127.29.219.67:45693: session id = 3e974c03d3c546f9ab6c91e762a5426b-b2619f3bed304d3ab864f56d5b655574
I20240501 20:10:29.516940 7175 tablet_copy_source_session.cc:205] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:29.524315 10931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2619f3bed304d3ab864f56d5b655574. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:29.551280 10931 tablet_copy_client.cc:783] T b2619f3bed304d3ab864f56d5b655574 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:29.551597 10931 tablet_copy_client.cc:647] T b2619f3bed304d3ab864f56d5b655574 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:29.559183 10931 tablet_copy_client.cc:523] T b2619f3bed304d3ab864f56d5b655574 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:29.563864 7152 raft_consensus.cc:1203] T 65b0048212a745c08ab24ead346bf74e P db0742ee3860468590eefef6ae6074fd [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240501 20:10:29.567523 11099 tablet_bootstrap.cc:492] T ca4f195a6dd14be981395538c2a33ad9 P db0742ee3860468590eefef6ae6074fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:29.568063 11099 tablet_bootstrap.cc:492] T ca4f195a6dd14be981395538c2a33ad9 P db0742ee3860468590eefef6ae6074fd: Bootstrap complete.
I20240501 20:10:29.568394 11099 ts_tablet_manager.cc:1367] T ca4f195a6dd14be981395538c2a33ad9 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.757s user 0.068s sys 0.010s
I20240501 20:10:29.571923 10931 tablet_bootstrap.cc:492] T b2619f3bed304d3ab864f56d5b655574 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:29.582748 11099 raft_consensus.cc:348] T ca4f195a6dd14be981395538c2a33ad9 P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:29.583097 11099 raft_consensus.cc:729] T ca4f195a6dd14be981395538c2a33ad9 P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: LEARNER
I20240501 20:10:29.583317 11099 consensus_queue.cc:260] T ca4f195a6dd14be981395538c2a33ad9 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:29.583966 11099 ts_tablet_manager.cc:1398] T ca4f195a6dd14be981395538c2a33ad9 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:29.586273 4519 tablet_copy_service.cc:342] P 3e974c03d3c546f9ab6c91e762a5426b: Request end of tablet copy session db0742ee3860468590eefef6ae6074fd-ca4f195a6dd14be981395538c2a33ad9 received from {username='slave'} at 127.29.219.70:56941
I20240501 20:10:29.586421 4519 tablet_copy_service.cc:433] P 3e974c03d3c546f9ab6c91e762a5426b: ending tablet copy session db0742ee3860468590eefef6ae6074fd-ca4f195a6dd14be981395538c2a33ad9 on tablet ca4f195a6dd14be981395538c2a33ad9 with peer db0742ee3860468590eefef6ae6074fd
I20240501 20:10:29.625319 11229 ts_tablet_manager.cc:902] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530: Initiating tablet copy from peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:29.627429 10931 tablet_bootstrap.cc:492] T b2619f3bed304d3ab864f56d5b655574 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:29.628005 10931 tablet_bootstrap.cc:492] T b2619f3bed304d3ab864f56d5b655574 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap complete.
I20240501 20:10:29.628276 10931 ts_tablet_manager.cc:1367] T b2619f3bed304d3ab864f56d5b655574 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.056s user 0.007s sys 0.002s
I20240501 20:10:29.629285 10931 raft_consensus.cc:348] T b2619f3bed304d3ab864f56d5b655574 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:29.629570 10931 raft_consensus.cc:729] T b2619f3bed304d3ab864f56d5b655574 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: LEARNER
I20240501 20:10:29.629770 10931 consensus_queue.cc:260] T b2619f3bed304d3ab864f56d5b655574 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:29.642740 10931 ts_tablet_manager.cc:1398] T b2619f3bed304d3ab864f56d5b655574 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.014s user 0.002s sys 0.001s
I20240501 20:10:29.643465 7175 tablet_copy_service.cc:342] P db0742ee3860468590eefef6ae6074fd: Request end of tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-b2619f3bed304d3ab864f56d5b655574 received from {username='slave'} at 127.29.219.67:45693
I20240501 20:10:29.643584 7175 tablet_copy_service.cc:433] P db0742ee3860468590eefef6ae6074fd: ending tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-b2619f3bed304d3ab864f56d5b655574 on tablet b2619f3bed304d3ab864f56d5b655574 with peer 3e974c03d3c546f9ab6c91e762a5426b
I20240501 20:10:29.665618 7019 raft_consensus.cc:1203] T 2aa1a5f266e840698f2127289628dcba P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:10:29.715133 11229 tablet_copy_client.cc:308] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.69:46237
I20240501 20:10:29.716154 7042 tablet_copy_service.cc:140] P ef0a4ada548446b3a531cb39a3c4e38d: Received BeginTabletCopySession request for tablet 1d860bb204134f0994462275519feecf from peer dca57823003e4729b769ca12367d7530 ({username='slave'} at 127.29.219.65:40207)
I20240501 20:10:29.716328 7042 tablet_copy_service.cc:161] P ef0a4ada548446b3a531cb39a3c4e38d: Beginning new tablet copy session on tablet 1d860bb204134f0994462275519feecf from peer dca57823003e4729b769ca12367d7530 at {username='slave'} at 127.29.219.65:40207: session id = dca57823003e4729b769ca12367d7530-1d860bb204134f0994462275519feecf
I20240501 20:10:29.723030 7042 tablet_copy_source_session.cc:205] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:29.727634 11229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d860bb204134f0994462275519feecf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:29.737241 10871 raft_consensus.cc:1050] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd: attempting to promote NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b to VOTER
I20240501 20:10:29.739096 10871 consensus_queue.cc:237] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:29.754143 11229 tablet_copy_client.cc:783] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:29.756971 11229 tablet_copy_client.cc:647] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:29.759395 11229 tablet_copy_client.cc:523] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:29.776549 4499 raft_consensus.cc:1261] T 5215e3f2040747f29aa1c9181a628096 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:29.776962 7287 raft_consensus.cc:1261] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:29.777808 10871 consensus_queue.cc:1035] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:29.779145 11016 consensus_queue.cc:1035] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:29.780311 4232 raft_consensus.cc:1261] T 5215e3f2040747f29aa1c9181a628096 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 20:10:29.782115 11016 consensus_queue.cc:1035] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:10:29.790549 11229 tablet_bootstrap.cc:492] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:29.791863 10871 raft_consensus.cc:2932] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:29.797605 4499 raft_consensus.cc:2932] T 5215e3f2040747f29aa1c9181a628096 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:29.802937 4232 raft_consensus.cc:2932] T 5215e3f2040747f29aa1c9181a628096 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:29.806082 4084 catalog_manager.cc:5484] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 5 to 6, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:29.817916 7153 consensus_queue.cc:237] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:29.826416 4232 raft_consensus.cc:1261] T 5215e3f2040747f29aa1c9181a628096 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:29.829191 11229 tablet_bootstrap.cc:492] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:29.829581 11229 tablet_bootstrap.cc:492] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530: Bootstrap complete.
I20240501 20:10:29.829828 11229 ts_tablet_manager.cc:1367] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.005s
I20240501 20:10:29.830549 11229 raft_consensus.cc:348] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:29.830807 11229 raft_consensus.cc:729] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: LEARNER
I20240501 20:10:29.830992 11229 consensus_queue.cc:260] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:29.831175 4496 raft_consensus.cc:1261] T 5215e3f2040747f29aa1c9181a628096 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:29.831524 11229 ts_tablet_manager.cc:1398] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:29.834131 11016 consensus_queue.cc:1035] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:29.838740 7042 tablet_copy_service.cc:342] P ef0a4ada548446b3a531cb39a3c4e38d: Request end of tablet copy session dca57823003e4729b769ca12367d7530-1d860bb204134f0994462275519feecf received from {username='slave'} at 127.29.219.65:40207
I20240501 20:10:29.838955 7042 tablet_copy_service.cc:433] P ef0a4ada548446b3a531cb39a3c4e38d: ending tablet copy session dca57823003e4729b769ca12367d7530-1d860bb204134f0994462275519feecf on tablet 1d860bb204134f0994462275519feecf with peer dca57823003e4729b769ca12367d7530
I20240501 20:10:29.841979 7153 raft_consensus.cc:1203] T ca4f195a6dd14be981395538c2a33ad9 P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:10:29.843503 11113 consensus_queue.cc:1035] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:29.850554 11113 raft_consensus.cc:2932] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:29.864979 4496 raft_consensus.cc:2932] T 5215e3f2040747f29aa1c9181a628096 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:29.881826 4084 catalog_manager.cc:5484] T 5215e3f2040747f29aa1c9181a628096 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 6 to 7, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:29.885782 7267 tablet_service.cc:1514] Processing DeleteTablet for tablet 5215e3f2040747f29aa1c9181a628096 with delete_type TABLET_DATA_TOMBSTONED (TS bceeae1c79964acabdae8ff771bf0c97 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54166
W20240501 20:10:29.915185 4301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.69:46237, user_credentials={real_user=slave}} blocked reactor thread for 46069.7us
I20240501 20:10:29.917321 11252 tablet_replica.cc:330] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:29.917636 11252 raft_consensus.cc:2229] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:29.917867 11252 pending_rounds.cc:62] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97: Trying to abort 1 pending ops.
I20240501 20:10:29.917930 11252 pending_rounds.cc:68] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7022977965008334848 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "5215e3f2040747f29aa1c9181a628096" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } } }
I20240501 20:10:29.918349 11252 raft_consensus.cc:2894] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240501 20:10:29.918473 11252 raft_consensus.cc:2866] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:10:29.918783 11252 raft_consensus.cc:2258] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:29.923542 11252 ts_tablet_manager.cc:1875] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:29.926450 4076 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5215e3f2040747f29aa1c9181a628096 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:29.927740 4232 raft_consensus.cc:2932] T 5215e3f2040747f29aa1c9181a628096 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:29.935101 11252 ts_tablet_manager.cc:1888] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 20:10:29.935261 11252 log.cc:1196] T 5215e3f2040747f29aa1c9181a628096 P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/5215e3f2040747f29aa1c9181a628096
I20240501 20:10:29.936612 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet 5215e3f2040747f29aa1c9181a628096 (table rebalancer_extra_table_0 [id=40f3268f46a14344bdf5de27989a6f1a]) successfully deleted
I20240501 20:10:30.022950 11118 raft_consensus.cc:1050] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e: attempting to promote NON_VOTER db0742ee3860468590eefef6ae6074fd to VOTER
I20240501 20:10:30.023715 11118 consensus_queue.cc:237] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:30.034459 7283 raft_consensus.cc:1261] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:10:30.034667 7153 raft_consensus.cc:1261] T 65b0048212a745c08ab24ead346bf74e P db0742ee3860468590eefef6ae6074fd [term 2 LEARNER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:10:30.035172 4365 raft_consensus.cc:1261] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a [term 2 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:10:30.035305 10377 consensus_queue.cc:1035] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:30.035885 10377 consensus_queue.cc:1035] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:30.036722 11221 consensus_queue.cc:1035] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:30.043632 11221 raft_consensus.cc:2932] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:30.047412 4363 raft_consensus.cc:2932] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:30.051945 7154 raft_consensus.cc:2932] T 65b0048212a745c08ab24ead346bf74e P db0742ee3860468590eefef6ae6074fd [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:30.053128 7283 raft_consensus.cc:2932] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:30.055114 4084 catalog_manager.cc:5484] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 8 to 9, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:30.060249 4628 consensus_queue.cc:237] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:30.069555 7153 raft_consensus.cc:1261] T 65b0048212a745c08ab24ead346bf74e P db0742ee3860468590eefef6ae6074fd [term 2 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:10:30.070374 11221 consensus_queue.cc:1035] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:10:30.074577 10377 raft_consensus.cc:2932] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:30.075237 7153 raft_consensus.cc:2932] T 65b0048212a745c08ab24ead346bf74e P db0742ee3860468590eefef6ae6074fd [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:30.078047 7287 raft_consensus.cc:1261] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:10:30.079567 11118 consensus_queue.cc:1035] T 65b0048212a745c08ab24ead346bf74e P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:10:30.081187 7283 raft_consensus.cc:2932] T 65b0048212a745c08ab24ead346bf74e P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:30.083101 4232 raft_consensus.cc:1203] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:30.087029 4499 raft_consensus.cc:1203] T b2619f3bed304d3ab864f56d5b655574 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:30.087659 4079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 65b0048212a745c08ab24ead346bf74e with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:30.088124 4087 catalog_manager.cc:5484] T 65b0048212a745c08ab24ead346bf74e P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 9 to 10, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New cstate: current_term: 2 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:30.092775 4346 tablet_service.cc:1514] Processing DeleteTablet for tablet 65b0048212a745c08ab24ead346bf74e with delete_type TABLET_DATA_TOMBSTONED (TS 10b47782558c41ee91f5e164a372049a not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:30.107030 11281 tablet_replica.cc:330] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:30.107327 11281 raft_consensus.cc:2229] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:30.107564 11281 raft_consensus.cc:2258] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:30.108659 11281 ts_tablet_manager.cc:1875] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:30.123481 11281 ts_tablet_manager.cc:1888] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 20:10:30.123613 11281 log.cc:1196] T 65b0048212a745c08ab24ead346bf74e P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/65b0048212a745c08ab24ead346bf74e
I20240501 20:10:30.124495 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet 65b0048212a745c08ab24ead346bf74e (table rebalance_test_table_9 [id=eef6790ef9874038b63d81f758f188d7]) successfully deleted
I20240501 20:10:30.198817 7287 consensus_queue.cc:237] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
W20240501 20:10:30.201382 7222 consensus_peers.cc:485] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 965b65ec896648698fbf3e1266d8e678. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:30.201794 7022 raft_consensus.cc:1261] T 965b65ec896648698fbf3e1266d8e678 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:30.202476 10940 consensus_queue.cc:1035] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:30.202569 4499 raft_consensus.cc:1261] T 965b65ec896648698fbf3e1266d8e678 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:30.203377 10940 consensus_queue.cc:1035] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:30.220708 10940 raft_consensus.cc:2932] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:30.221944 7022 raft_consensus.cc:2932] T 965b65ec896648698fbf3e1266d8e678 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:30.222064 4499 raft_consensus.cc:2932] T 965b65ec896648698fbf3e1266d8e678 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:30.229960 4087 catalog_manager.cc:5484] T 965b65ec896648698fbf3e1266d8e678 P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:30.236765 11049 raft_consensus.cc:1050] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b: attempting to promote NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d to VOTER
I20240501 20:10:30.237730 11049 consensus_queue.cc:237] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:30.244731 7022 raft_consensus.cc:1261] T 2aa1a5f266e840698f2127289628dcba P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:30.244897 7420 raft_consensus.cc:1261] T 2aa1a5f266e840698f2127289628dcba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:30.245463 11047 consensus_queue.cc:1035] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:30.246477 4363 raft_consensus.cc:1261] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:30.247660 11050 consensus_queue.cc:1035] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:30.248193 11050 consensus_queue.cc:1035] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:30.265498 11047 raft_consensus.cc:2932] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:30.271366 7420 raft_consensus.cc:2932] T 2aa1a5f266e840698f2127289628dcba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:30.274354 7022 raft_consensus.cc:2932] T 2aa1a5f266e840698f2127289628dcba P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:30.277662 4365 raft_consensus.cc:2932] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:30.281327 4088 catalog_manager.cc:5484] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 7 to 8, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:30.292508 4499 consensus_queue.cc:237] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:30.294708 7022 raft_consensus.cc:1261] T 2aa1a5f266e840698f2127289628dcba P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:30.295251 11047 consensus_queue.cc:1035] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:30.296032 7416 raft_consensus.cc:1261] T 2aa1a5f266e840698f2127289628dcba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:30.298542 11047 consensus_queue.cc:1035] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240501 20:10:30.306433 11050 raft_consensus.cc:2932] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:30.309700 4499 consensus_queue.cc:237] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:30.310557 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2aa1a5f266e840698f2127289628dcba with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:30.315977 7019 raft_consensus.cc:2932] T 2aa1a5f266e840698f2127289628dcba P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:30.320006 7420 raft_consensus.cc:2932] T 2aa1a5f266e840698f2127289628dcba P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
W20240501 20:10:30.326714 4433 consensus_peers.cc:485] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b -> Peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Couldn't send request to peer bceeae1c79964acabdae8ff771bf0c97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffaf0a0105ea4e1f9ed9d90c221ebc8a. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:30.314182 4088 catalog_manager.cc:5484] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 8 to 9, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:30.332945 4365 raft_consensus.cc:1261] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:30.338085 7416 raft_consensus.cc:1261] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:30.338227 11049 consensus_queue.cc:1035] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:30.338744 11047 consensus_queue.cc:1035] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:30.341245 4346 tablet_service.cc:1514] Processing DeleteTablet for tablet 2aa1a5f266e840698f2127289628dcba with delete_type TABLET_DATA_TOMBSTONED (TS 10b47782558c41ee91f5e164a372049a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:30.341578 11281 tablet_replica.cc:330] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:30.341867 11281 raft_consensus.cc:2229] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:30.342258 11281 raft_consensus.cc:2258] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:30.343523 11281 ts_tablet_manager.cc:1875] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:30.353005 11281 ts_tablet_manager.cc:1888] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:10:30.354154 11281 log.cc:1196] T 2aa1a5f266e840698f2127289628dcba P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/2aa1a5f266e840698f2127289628dcba
I20240501 20:10:30.355144 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet 2aa1a5f266e840698f2127289628dcba (table rebalance_test_table_7 [id=59dd8401cbd048cea30daf86baac8cfa]) successfully deleted
I20240501 20:10:30.360046 11324 raft_consensus.cc:2932] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:30.361810 4365 raft_consensus.cc:2932] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:30.366935 4089 catalog_manager.cc:5484] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 6 to 7, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:30.376273 7416 raft_consensus.cc:2932] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:30.395998 4365 consensus_queue.cc:237] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:30.399849 7287 raft_consensus.cc:1261] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:30.401140 4499 raft_consensus.cc:1261] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:30.401197 11251 consensus_queue.cc:1035] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 20:10:30.401682 4301 consensus_peers.cc:485] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:30.402017 11251 consensus_queue.cc:1035] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:30.409051 11251 raft_consensus.cc:2932] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:30.412906 4499 raft_consensus.cc:2932] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:30.416355 4089 catalog_manager.cc:5484] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index 6 to 7, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:30.427228 7287 raft_consensus.cc:2932] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:30.442523 11047 raft_consensus.cc:1050] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b: attempting to promote NON_VOTER db0742ee3860468590eefef6ae6074fd to VOTER
I20240501 20:10:30.443363 11047 consensus_queue.cc:237] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:30.445451 4234 raft_consensus.cc:1261] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:30.445480 7287 raft_consensus.cc:1261] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:30.446187 11324 consensus_queue.cc:1035] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:30.447436 7154 raft_consensus.cc:1261] T ca4f195a6dd14be981395538c2a33ad9 P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:30.447935 11324 consensus_queue.cc:1035] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:30.448951 11324 consensus_queue.cc:1035] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:30.451503 4628 consensus_queue.cc:237] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:30.455217 11050 raft_consensus.cc:2932] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:30.456158 7287 raft_consensus.cc:2932] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:30.460222 4234 raft_consensus.cc:2932] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
W20240501 20:10:30.463909 4564 consensus_peers.cc:485] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25f01a01106f47c489e252c928cf28bb. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:30.464015 7154 raft_consensus.cc:1261] T 25f01a01106f47c489e252c928cf28bb P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:30.464183 4088 catalog_manager.cc:5484] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index 7 to 8, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:30.464565 11118 consensus_queue.cc:1035] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:30.465314 7154 raft_consensus.cc:2932] T ca4f195a6dd14be981395538c2a33ad9 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:30.469769 7287 raft_consensus.cc:1261] T 25f01a01106f47c489e252c928cf28bb P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:30.470325 11118 consensus_queue.cc:1035] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:30.472388 10377 raft_consensus.cc:2932] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:30.473181 7154 raft_consensus.cc:2932] T 25f01a01106f47c489e252c928cf28bb P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:30.476907 4496 consensus_queue.cc:237] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:30.479338 7283 raft_consensus.cc:1261] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:30.480470 11324 consensus_queue.cc:1035] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:30.480648 7154 raft_consensus.cc:1261] T ca4f195a6dd14be981395538c2a33ad9 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:30.481160 11324 consensus_queue.cc:1035] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:30.482265 4089 catalog_manager.cc:5484] T 25f01a01106f47c489e252c928cf28bb P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index -1 to 2, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:30.492241 11324 raft_consensus.cc:2932] T ca4f195a6dd14be981395538c2a33ad9 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:30.493345 7283 raft_consensus.cc:2932] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:30.496201 7153 raft_consensus.cc:2932] T ca4f195a6dd14be981395538c2a33ad9 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:30.498591 4088 catalog_manager.cc:5484] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index 8 to 9, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:30.502045 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ca4f195a6dd14be981395538c2a33ad9 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:30.502610 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet ca4f195a6dd14be981395538c2a33ad9 with delete_type TABLET_DATA_TOMBSTONED (TS dca57823003e4729b769ca12367d7530 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:30.504601 11137 raft_consensus.cc:1050] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d: attempting to promote NON_VOTER dca57823003e4729b769ca12367d7530 to VOTER
I20240501 20:10:30.505421 11351 tablet_replica.cc:330] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:30.505621 11351 raft_consensus.cc:2229] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:30.505504 11137 consensus_queue.cc:237] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:30.505968 11351 raft_consensus.cc:2258] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:30.506943 11351 ts_tablet_manager.cc:1875] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:30.507668 7287 raft_consensus.cc:2932] T 25f01a01106f47c489e252c928cf28bb P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:30.515199 11351 ts_tablet_manager.cc:1888] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:10:30.515302 11351 log.cc:1196] T ca4f195a6dd14be981395538c2a33ad9 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/ca4f195a6dd14be981395538c2a33ad9
I20240501 20:10:30.515949 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet ca4f195a6dd14be981395538c2a33ad9 (table rebalance_test_table_19 [id=8de1e355c457486da8bba4c628d52c17]) successfully deleted
I20240501 20:10:30.531914 4234 raft_consensus.cc:1261] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:30.532158 7153 raft_consensus.cc:1261] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:30.532266 4496 raft_consensus.cc:1261] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:30.532778 11111 consensus_queue.cc:1035] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:30.533329 11354 consensus_queue.cc:1035] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:30.533888 11137 consensus_queue.cc:1035] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:30.536551 11113 raft_consensus.cc:1050] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd: attempting to promote NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b to VOTER
I20240501 20:10:30.537632 11113 consensus_queue.cc:237] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:30.546905 4498 raft_consensus.cc:1261] T b2619f3bed304d3ab864f56d5b655574 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:30.547611 11250 consensus_queue.cc:1035] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:30.549746 11137 raft_consensus.cc:2932] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:30.550869 4628 raft_consensus.cc:1261] T b2619f3bed304d3ab864f56d5b655574 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:30.551884 11250 consensus_queue.cc:1035] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:30.557418 7287 raft_consensus.cc:1261] T b2619f3bed304d3ab864f56d5b655574 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:30.558871 11113 consensus_queue.cc:1035] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:30.573348 7153 raft_consensus.cc:2932] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:30.579841 11113 raft_consensus.cc:2932] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:30.584419 4089 catalog_manager.cc:5484] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:30.587448 4498 raft_consensus.cc:2932] T b2619f3bed304d3ab864f56d5b655574 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:30.616447 7281 raft_consensus.cc:2932] T b2619f3bed304d3ab864f56d5b655574 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:30.618979 4087 catalog_manager.cc:5484] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:30.634691 7153 consensus_queue.cc:237] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:30.637850 4627 raft_consensus.cc:1261] T b2619f3bed304d3ab864f56d5b655574 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:30.638749 11016 consensus_queue.cc:1035] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:30.640276 4627 raft_consensus.cc:2932] T b2619f3bed304d3ab864f56d5b655574 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:30.652617 4498 raft_consensus.cc:1261] T b2619f3bed304d3ab864f56d5b655574 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:30.653405 11016 consensus_queue.cc:1035] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:30.664561 11016 raft_consensus.cc:2932] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:30.673350 4076 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b2619f3bed304d3ab864f56d5b655574 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:30.673350 4498 raft_consensus.cc:2932] T b2619f3bed304d3ab864f56d5b655574 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:30.677314 4089 catalog_manager.cc:5484] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:30.680282 4627 raft_consensus.cc:2932] T b2619f3bed304d3ab864f56d5b655574 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:30.684129 11364 ts_tablet_manager.cc:902] T 965b65ec896648698fbf3e1266d8e678 P 10b47782558c41ee91f5e164a372049a: Initiating tablet copy from peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:30.688576 7267 tablet_service.cc:1514] Processing DeleteTablet for tablet b2619f3bed304d3ab864f56d5b655574 with delete_type TABLET_DATA_TOMBSTONED (TS bceeae1c79964acabdae8ff771bf0c97 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54166
W20240501 20:10:30.730408 6957 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.67:34595, user_credentials={real_user=slave}} blocked reactor thread for 182900us
I20240501 20:10:30.736450 4498 raft_consensus.cc:2932] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:30.743718 4232 raft_consensus.cc:2932] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:30.751890 7019 consensus_queue.cc:237] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:30.762995 4498 raft_consensus.cc:1261] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:30.763613 4232 raft_consensus.cc:1261] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:30.767181 11367 tablet_replica.cc:330] T b2619f3bed304d3ab864f56d5b655574 P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:30.767441 11367 raft_consensus.cc:2229] T b2619f3bed304d3ab864f56d5b655574 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:30.767745 11367 raft_consensus.cc:2258] T b2619f3bed304d3ab864f56d5b655574 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:30.768616 11359 consensus_queue.cc:1035] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20240501 20:10:30.773762 11363 consensus_queue.cc:1035] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:30.777323 11367 ts_tablet_manager.cc:1875] T b2619f3bed304d3ab864f56d5b655574 P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:30.779335 11111 raft_consensus.cc:2932] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:30.783229 11364 tablet_copy_client.cc:308] T 965b65ec896648698fbf3e1266d8e678 P 10b47782558c41ee91f5e164a372049a: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.71:39447
I20240501 20:10:30.784046 7308 tablet_copy_service.cc:140] P bceeae1c79964acabdae8ff771bf0c97: Received BeginTabletCopySession request for tablet 965b65ec896648698fbf3e1266d8e678 from peer 10b47782558c41ee91f5e164a372049a ({username='slave'} at 127.29.219.66:32801)
I20240501 20:10:30.784230 7308 tablet_copy_service.cc:161] P bceeae1c79964acabdae8ff771bf0c97: Beginning new tablet copy session on tablet 965b65ec896648698fbf3e1266d8e678 from peer 10b47782558c41ee91f5e164a372049a at {username='slave'} at 127.29.219.66:32801: session id = 10b47782558c41ee91f5e164a372049a-965b65ec896648698fbf3e1266d8e678
I20240501 20:10:30.787518 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1d860bb204134f0994462275519feecf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:30.788226 7308 tablet_copy_source_session.cc:205] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:30.788461 4234 raft_consensus.cc:2932] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:30.789166 4498 raft_consensus.cc:2932] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:30.795120 11364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 965b65ec896648698fbf3e1266d8e678. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:30.798332 4089 catalog_manager.cc:5484] T 1d860bb204134f0994462275519feecf P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:30.805689 7135 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d860bb204134f0994462275519feecf with delete_type TABLET_DATA_TOMBSTONED (TS db0742ee3860468590eefef6ae6074fd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52036
I20240501 20:10:30.805830 11364 tablet_copy_client.cc:783] T 965b65ec896648698fbf3e1266d8e678 P 10b47782558c41ee91f5e164a372049a: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:30.806077 11364 tablet_copy_client.cc:647] T 965b65ec896648698fbf3e1266d8e678 P 10b47782558c41ee91f5e164a372049a: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:30.810369 11364 tablet_copy_client.cc:523] T 965b65ec896648698fbf3e1266d8e678 P 10b47782558c41ee91f5e164a372049a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:30.822319 11364 tablet_bootstrap.cc:492] T 965b65ec896648698fbf3e1266d8e678 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:30.853843 11367 ts_tablet_manager.cc:1888] T b2619f3bed304d3ab864f56d5b655574 P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:30.854027 11367 log.cc:1196] T b2619f3bed304d3ab864f56d5b655574 P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/b2619f3bed304d3ab864f56d5b655574
I20240501 20:10:30.855195 11371 tablet_replica.cc:330] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd: stopping tablet replica
I20240501 20:10:30.856739 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet b2619f3bed304d3ab864f56d5b655574 (table rebalancer_extra_table_26 [id=306aca4fdfa94f4f81f64f300d602707]) successfully deleted
I20240501 20:10:30.859184 11364 tablet_bootstrap.cc:492] T 965b65ec896648698fbf3e1266d8e678 P 10b47782558c41ee91f5e164a372049a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:30.859552 11364 tablet_bootstrap.cc:492] T 965b65ec896648698fbf3e1266d8e678 P 10b47782558c41ee91f5e164a372049a: Bootstrap complete.
I20240501 20:10:30.859212 11371 raft_consensus.cc:2229] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:30.859784 11364 ts_tablet_manager.cc:1367] T 965b65ec896648698fbf3e1266d8e678 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.000s
I20240501 20:10:30.860116 11371 raft_consensus.cc:2258] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:30.860469 11364 raft_consensus.cc:348] T 965b65ec896648698fbf3e1266d8e678 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:30.860684 11364 raft_consensus.cc:729] T 965b65ec896648698fbf3e1266d8e678 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: LEARNER
I20240501 20:10:30.860852 11364 consensus_queue.cc:260] T 965b65ec896648698fbf3e1266d8e678 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:30.861444 11371 ts_tablet_manager.cc:1875] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:30.862748 11364 ts_tablet_manager.cc:1398] T 965b65ec896648698fbf3e1266d8e678 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:10:30.865136 7308 tablet_copy_service.cc:342] P bceeae1c79964acabdae8ff771bf0c97: Request end of tablet copy session 10b47782558c41ee91f5e164a372049a-965b65ec896648698fbf3e1266d8e678 received from {username='slave'} at 127.29.219.66:32801
I20240501 20:10:30.865291 7308 tablet_copy_service.cc:433] P bceeae1c79964acabdae8ff771bf0c97: ending tablet copy session 10b47782558c41ee91f5e164a372049a-965b65ec896648698fbf3e1266d8e678 on tablet 965b65ec896648698fbf3e1266d8e678 with peer 10b47782558c41ee91f5e164a372049a
I20240501 20:10:30.868340 11371 ts_tablet_manager.cc:1888] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:30.868535 11371 log.cc:1196] T 1d860bb204134f0994462275519feecf P db0742ee3860468590eefef6ae6074fd: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/wals/1d860bb204134f0994462275519feecf
I20240501 20:10:30.869756 4076 catalog_manager.cc:4830] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): tablet 1d860bb204134f0994462275519feecf (table rebalancer_extra_table_16 [id=514d975194074c92815ac14399279d97]) successfully deleted
I20240501 20:10:30.883668 11376 ts_tablet_manager.cc:902] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97: Initiating tablet copy from peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:30.890144 7420 consensus_queue.cc:237] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:30.893018 7153 raft_consensus.cc:1261] T 537b5ca846b5485ca1d98351d9038430 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:30.893172 7355 consensus_peers.cc:485] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 537b5ca846b5485ca1d98351d9038430. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:30.894165 4627 raft_consensus.cc:1261] T 537b5ca846b5485ca1d98351d9038430 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:30.893839 11101 consensus_queue.cc:1035] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240501 20:10:30.896939 11101 consensus_queue.cc:1035] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:30.901958 11101 raft_consensus.cc:2932] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:30.903797 11383 ts_tablet_manager.cc:902] T 25f01a01106f47c489e252c928cf28bb P ef0a4ada548446b3a531cb39a3c4e38d: Initiating tablet copy from peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:30.909665 4628 raft_consensus.cc:2932] T 537b5ca846b5485ca1d98351d9038430 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:30.913760 11383 tablet_copy_client.cc:308] T 25f01a01106f47c489e252c928cf28bb P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.68:45205
I20240501 20:10:30.918262 4651 tablet_copy_service.cc:140] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Received BeginTabletCopySession request for tablet 25f01a01106f47c489e252c928cf28bb from peer ef0a4ada548446b3a531cb39a3c4e38d ({username='slave'} at 127.29.219.69:59565)
I20240501 20:10:30.918462 4651 tablet_copy_service.cc:161] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Beginning new tablet copy session on tablet 25f01a01106f47c489e252c928cf28bb from peer ef0a4ada548446b3a531cb39a3c4e38d at {username='slave'} at 127.29.219.69:59565: session id = ef0a4ada548446b3a531cb39a3c4e38d-25f01a01106f47c489e252c928cf28bb
I20240501 20:10:30.918648 7153 raft_consensus.cc:2932] T 537b5ca846b5485ca1d98351d9038430 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:30.922480 4651 tablet_copy_source_session.cc:205] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:30.930341 4087 catalog_manager.cc:5484] T 537b5ca846b5485ca1d98351d9038430 P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:30.931476 11388 ts_tablet_manager.cc:902] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530: Initiating tablet copy from peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:30.931571 11383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25f01a01106f47c489e252c928cf28bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:30.945581 11383 tablet_copy_client.cc:783] T 25f01a01106f47c489e252c928cf28bb P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:30.946471 11383 tablet_copy_client.cc:647] T 25f01a01106f47c489e252c928cf28bb P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:30.952207 11376 tablet_copy_client.cc:308] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.67:34595
I20240501 20:10:30.952980 4519 tablet_copy_service.cc:140] P 3e974c03d3c546f9ab6c91e762a5426b: Received BeginTabletCopySession request for tablet ffaf0a0105ea4e1f9ed9d90c221ebc8a from peer bceeae1c79964acabdae8ff771bf0c97 ({username='slave'} at 127.29.219.71:60861)
I20240501 20:10:30.953143 4519 tablet_copy_service.cc:161] P 3e974c03d3c546f9ab6c91e762a5426b: Beginning new tablet copy session on tablet ffaf0a0105ea4e1f9ed9d90c221ebc8a from peer bceeae1c79964acabdae8ff771bf0c97 at {username='slave'} at 127.29.219.71:60861: session id = bceeae1c79964acabdae8ff771bf0c97-ffaf0a0105ea4e1f9ed9d90c221ebc8a
I20240501 20:10:30.955794 4519 tablet_copy_source_session.cc:205] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:30.958604 11383 tablet_copy_client.cc:523] T 25f01a01106f47c489e252c928cf28bb P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:30.959823 11376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffaf0a0105ea4e1f9ed9d90c221ebc8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:30.966255 11376 tablet_copy_client.cc:783] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:30.966544 11376 tablet_copy_client.cc:647] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:30.968394 11383 tablet_bootstrap.cc:492] T 25f01a01106f47c489e252c928cf28bb P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:30.970369 11376 tablet_copy_client.cc:523] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:30.974357 11376 tablet_bootstrap.cc:492] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:30.977187 11388 tablet_copy_client.cc:308] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.66:40689
I20240501 20:10:30.979066 4388 tablet_copy_service.cc:140] P 10b47782558c41ee91f5e164a372049a: Received BeginTabletCopySession request for tablet f40b062e103d4e54a913fa0f4111f623 from peer dca57823003e4729b769ca12367d7530 ({username='slave'} at 127.29.219.65:41491)
I20240501 20:10:30.979209 4388 tablet_copy_service.cc:161] P 10b47782558c41ee91f5e164a372049a: Beginning new tablet copy session on tablet f40b062e103d4e54a913fa0f4111f623 from peer dca57823003e4729b769ca12367d7530 at {username='slave'} at 127.29.219.65:41491: session id = dca57823003e4729b769ca12367d7530-f40b062e103d4e54a913fa0f4111f623
I20240501 20:10:30.980048 11383 tablet_bootstrap.cc:492] T 25f01a01106f47c489e252c928cf28bb P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:30.980526 11383 tablet_bootstrap.cc:492] T 25f01a01106f47c489e252c928cf28bb P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap complete.
I20240501 20:10:30.980767 11383 ts_tablet_manager.cc:1367] T 25f01a01106f47c489e252c928cf28bb P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20240501 20:10:30.981488 11383 raft_consensus.cc:348] T 25f01a01106f47c489e252c928cf28bb P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:30.981766 11383 raft_consensus.cc:729] T 25f01a01106f47c489e252c928cf28bb P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: LEARNER
I20240501 20:10:30.981988 4388 tablet_copy_source_session.cc:205] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:30.981981 11383 consensus_queue.cc:260] T 25f01a01106f47c489e252c928cf28bb P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:30.983098 11388 ts_tablet_manager.cc:1875] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:10:30.984030 11383 ts_tablet_manager.cc:1398] T 25f01a01106f47c489e252c928cf28bb P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:30.986809 4651 tablet_copy_service.cc:342] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Request end of tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-25f01a01106f47c489e252c928cf28bb received from {username='slave'} at 127.29.219.69:59565
I20240501 20:10:30.986927 4651 tablet_copy_service.cc:433] P 1d34e18a1d394d3f91c9f50d1c7bea4e: ending tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-25f01a01106f47c489e252c928cf28bb on tablet 25f01a01106f47c489e252c928cf28bb with peer ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:30.989732 11388 ts_tablet_manager.cc:1888] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 20:10:30.990103 11388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f40b062e103d4e54a913fa0f4111f623. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:30.994279 11388 tablet_copy_client.cc:783] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:30.994534 11388 tablet_copy_client.cc:647] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:30.998785 11388 tablet_copy_client.cc:523] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:31.002213 11388 tablet_bootstrap.cc:492] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:31.009877 4498 consensus_queue.cc:237] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
W20240501 20:10:31.020133 4433 consensus_peers.cc:485] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b -> Peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Couldn't send request to peer bceeae1c79964acabdae8ff771bf0c97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46df0faa681b469abe9e383b181a6053. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:31.020233 7022 raft_consensus.cc:1261] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:31.020757 11050 consensus_queue.cc:1035] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:31.020854 4628 raft_consensus.cc:1261] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:31.021584 11050 consensus_queue.cc:1035] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:31.024008 11324 raft_consensus.cc:2932] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:31.024798 7022 raft_consensus.cc:2932] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:31.027503 4628 raft_consensus.cc:2932] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:31.033803 4089 catalog_manager.cc:5484] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 6 to 7, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:31.055459 11376 tablet_bootstrap.cc:492] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:31.055931 11376 tablet_bootstrap.cc:492] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97: Bootstrap complete.
I20240501 20:10:31.056201 11376 ts_tablet_manager.cc:1367] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.082s user 0.045s sys 0.012s
I20240501 20:10:31.056988 11376 raft_consensus.cc:348] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:31.057251 11376 raft_consensus.cc:729] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: LEARNER
I20240501 20:10:31.057474 11376 consensus_queue.cc:260] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:31.058442 11376 ts_tablet_manager.cc:1398] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 20:10:31.059373 11388 tablet_bootstrap.cc:492] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:31.059744 11388 tablet_bootstrap.cc:492] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530: Bootstrap complete.
I20240501 20:10:31.059973 11388 ts_tablet_manager.cc:1367] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.058s user 0.045s sys 0.011s
I20240501 20:10:31.060220 4519 tablet_copy_service.cc:342] P 3e974c03d3c546f9ab6c91e762a5426b: Request end of tablet copy session bceeae1c79964acabdae8ff771bf0c97-ffaf0a0105ea4e1f9ed9d90c221ebc8a received from {username='slave'} at 127.29.219.71:60861
I20240501 20:10:31.060317 4519 tablet_copy_service.cc:433] P 3e974c03d3c546f9ab6c91e762a5426b: ending tablet copy session bceeae1c79964acabdae8ff771bf0c97-ffaf0a0105ea4e1f9ed9d90c221ebc8a on tablet ffaf0a0105ea4e1f9ed9d90c221ebc8a with peer bceeae1c79964acabdae8ff771bf0c97
I20240501 20:10:31.060596 11388 raft_consensus.cc:348] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:31.060817 11388 raft_consensus.cc:729] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: LEARNER
I20240501 20:10:31.060992 11388 consensus_queue.cc:260] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:31.061574 11388 ts_tablet_manager.cc:1398] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:10:31.064666 4388 tablet_copy_service.cc:342] P 10b47782558c41ee91f5e164a372049a: Request end of tablet copy session dca57823003e4729b769ca12367d7530-f40b062e103d4e54a913fa0f4111f623 received from {username='slave'} at 127.29.219.65:41491
I20240501 20:10:31.064821 4388 tablet_copy_service.cc:433] P 10b47782558c41ee91f5e164a372049a: ending tablet copy session dca57823003e4729b769ca12367d7530-f40b062e103d4e54a913fa0f4111f623 on tablet f40b062e103d4e54a913fa0f4111f623 with peer dca57823003e4729b769ca12367d7530
I20240501 20:10:31.260689 4365 raft_consensus.cc:1203] T 965b65ec896648698fbf3e1266d8e678 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:31.280654 11364 ts_tablet_manager.cc:902] T 537b5ca846b5485ca1d98351d9038430 P 10b47782558c41ee91f5e164a372049a: Initiating tablet copy from peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:31.299458 11364 tablet_copy_client.cc:308] T 537b5ca846b5485ca1d98351d9038430 P 10b47782558c41ee91f5e164a372049a: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.72:39817
I20240501 20:10:31.300424 7441 tablet_copy_service.cc:140] P b9957d4086bc4a76b18ed9265a5739d5: Received BeginTabletCopySession request for tablet 537b5ca846b5485ca1d98351d9038430 from peer 10b47782558c41ee91f5e164a372049a ({username='slave'} at 127.29.219.66:55097)
I20240501 20:10:31.300640 7441 tablet_copy_service.cc:161] P b9957d4086bc4a76b18ed9265a5739d5: Beginning new tablet copy session on tablet 537b5ca846b5485ca1d98351d9038430 from peer 10b47782558c41ee91f5e164a372049a at {username='slave'} at 127.29.219.66:55097: session id = 10b47782558c41ee91f5e164a372049a-537b5ca846b5485ca1d98351d9038430
I20240501 20:10:31.300974 4628 consensus_queue.cc:237] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:31.304008 7441 tablet_copy_source_session.cc:205] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:31.305439 11364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 537b5ca846b5485ca1d98351d9038430. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 20:10:31.312147 4564 consensus_peers.cc:485] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04a5fb2b808d466491144e64eb54d0fd. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:31.312317 7154 raft_consensus.cc:1261] T 04a5fb2b808d466491144e64eb54d0fd P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:31.312886 11364 tablet_copy_client.cc:783] T 537b5ca846b5485ca1d98351d9038430 P 10b47782558c41ee91f5e164a372049a: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:31.312948 11118 consensus_queue.cc:1035] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:31.313172 11364 tablet_copy_client.cc:647] T 537b5ca846b5485ca1d98351d9038430 P 10b47782558c41ee91f5e164a372049a: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:31.314224 4365 raft_consensus.cc:1261] T 04a5fb2b808d466491144e64eb54d0fd P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:31.314806 11364 tablet_copy_client.cc:523] T 537b5ca846b5485ca1d98351d9038430 P 10b47782558c41ee91f5e164a372049a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:31.315300 11118 consensus_queue.cc:1035] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:31.320866 11364 tablet_bootstrap.cc:492] T 537b5ca846b5485ca1d98351d9038430 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:31.324879 11221 raft_consensus.cc:2932] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:31.328852 7153 raft_consensus.cc:2932] T 04a5fb2b808d466491144e64eb54d0fd P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:31.335796 11364 tablet_bootstrap.cc:492] T 537b5ca846b5485ca1d98351d9038430 P 10b47782558c41ee91f5e164a372049a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:31.336392 11364 tablet_bootstrap.cc:492] T 537b5ca846b5485ca1d98351d9038430 P 10b47782558c41ee91f5e164a372049a: Bootstrap complete.
I20240501 20:10:31.336774 11364 ts_tablet_manager.cc:1367] T 537b5ca846b5485ca1d98351d9038430 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20240501 20:10:31.337694 11364 raft_consensus.cc:348] T 537b5ca846b5485ca1d98351d9038430 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:31.338006 11364 raft_consensus.cc:729] T 537b5ca846b5485ca1d98351d9038430 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: LEARNER
I20240501 20:10:31.338224 11364 consensus_queue.cc:260] T 537b5ca846b5485ca1d98351d9038430 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:31.339037 11364 ts_tablet_manager.cc:1398] T 537b5ca846b5485ca1d98351d9038430 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:10:31.345443 4089 catalog_manager.cc:5484] T 04a5fb2b808d466491144e64eb54d0fd P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index -1 to 2, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:31.349625 7441 tablet_copy_service.cc:342] P b9957d4086bc4a76b18ed9265a5739d5: Request end of tablet copy session 10b47782558c41ee91f5e164a372049a-537b5ca846b5485ca1d98351d9038430 received from {username='slave'} at 127.29.219.66:55097
I20240501 20:10:31.349793 7441 tablet_copy_service.cc:433] P b9957d4086bc4a76b18ed9265a5739d5: ending tablet copy session 10b47782558c41ee91f5e164a372049a-537b5ca846b5485ca1d98351d9038430 on tablet 537b5ca846b5485ca1d98351d9038430 with peer 10b47782558c41ee91f5e164a372049a
I20240501 20:10:31.359678 4364 raft_consensus.cc:2932] T 04a5fb2b808d466491144e64eb54d0fd P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:31.373474 7022 raft_consensus.cc:1203] T 25f01a01106f47c489e252c928cf28bb P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:31.386906 4234 consensus_queue.cc:237] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:31.388936 7416 raft_consensus.cc:1261] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:10:31.388929 4498 raft_consensus.cc:1261] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 20:10:31.389398 4167 consensus_peers.cc:485] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 -> Peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Couldn't send request to peer db0742ee3860468590eefef6ae6074fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d690199665174c7d9bdbd54d2af4ac70. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:31.389998 11144 consensus_queue.cc:1035] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:31.390255 11146 consensus_queue.cc:1035] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 20:10:31.394829 11415 raft_consensus.cc:2932] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:31.397224 4498 raft_consensus.cc:2932] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:31.399049 7416 raft_consensus.cc:2932] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:31.401420 4087 catalog_manager.cc:5484] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 6 to 7, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:31.403182 7287 raft_consensus.cc:1203] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:10:31.440148 11376 ts_tablet_manager.cc:902] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97: Initiating tablet copy from peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:31.441354 11376 tablet_copy_client.cc:308] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.67:34595
I20240501 20:10:31.441933 4519 tablet_copy_service.cc:140] P 3e974c03d3c546f9ab6c91e762a5426b: Received BeginTabletCopySession request for tablet 46df0faa681b469abe9e383b181a6053 from peer bceeae1c79964acabdae8ff771bf0c97 ({username='slave'} at 127.29.219.71:60861)
I20240501 20:10:31.442090 4519 tablet_copy_service.cc:161] P 3e974c03d3c546f9ab6c91e762a5426b: Beginning new tablet copy session on tablet 46df0faa681b469abe9e383b181a6053 from peer bceeae1c79964acabdae8ff771bf0c97 at {username='slave'} at 127.29.219.71:60861: session id = bceeae1c79964acabdae8ff771bf0c97-46df0faa681b469abe9e383b181a6053
I20240501 20:10:31.444787 4519 tablet_copy_source_session.cc:205] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:31.446123 11376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46df0faa681b469abe9e383b181a6053. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:31.452914 11376 tablet_copy_client.cc:783] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:31.453161 11376 tablet_copy_client.cc:647] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:31.456637 11376 tablet_copy_client.cc:523] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:31.459889 11376 tablet_bootstrap.cc:492] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:31.467832 4234 raft_consensus.cc:1203] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:10:31.508090 11376 tablet_bootstrap.cc:492] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:31.508608 11376 tablet_bootstrap.cc:492] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap complete.
I20240501 20:10:31.508939 11376 ts_tablet_manager.cc:1367] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.049s user 0.038s sys 0.003s
I20240501 20:10:31.509809 11376 raft_consensus.cc:348] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:31.510067 11376 raft_consensus.cc:729] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: LEARNER
I20240501 20:10:31.510296 11376 consensus_queue.cc:260] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:31.512800 11376 ts_tablet_manager.cc:1398] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20240501 20:10:31.513490 4519 tablet_copy_service.cc:342] P 3e974c03d3c546f9ab6c91e762a5426b: Request end of tablet copy session bceeae1c79964acabdae8ff771bf0c97-46df0faa681b469abe9e383b181a6053 received from {username='slave'} at 127.29.219.71:60861
I20240501 20:10:31.513602 4519 tablet_copy_service.cc:433] P 3e974c03d3c546f9ab6c91e762a5426b: ending tablet copy session bceeae1c79964acabdae8ff771bf0c97-46df0faa681b469abe9e383b181a6053 on tablet 46df0faa681b469abe9e383b181a6053 with peer bceeae1c79964acabdae8ff771bf0c97
I20240501 20:10:31.794142 11311 raft_consensus.cc:1050] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97: attempting to promote NON_VOTER 10b47782558c41ee91f5e164a372049a to VOTER
I20240501 20:10:31.794973 11311 consensus_queue.cc:237] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } }
I20240501 20:10:31.805301 4364 raft_consensus.cc:1261] T 965b65ec896648698fbf3e1266d8e678 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:31.806135 7019 raft_consensus.cc:1261] T 965b65ec896648698fbf3e1266d8e678 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:31.806690 10940 consensus_queue.cc:1035] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 20:10:31.807528 10940 consensus_queue.cc:1035] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:31.813251 4498 raft_consensus.cc:1261] T 965b65ec896648698fbf3e1266d8e678 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:31.813942 11581 consensus_queue.cc:1035] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:31.817077 11581 raft_consensus.cc:2932] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:31.821849 11324 raft_consensus.cc:1050] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b: attempting to promote NON_VOTER bceeae1c79964acabdae8ff771bf0c97 to VOTER
I20240501 20:10:31.822682 11324 consensus_queue.cc:237] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:31.822796 4498 raft_consensus.cc:2932] T 965b65ec896648698fbf3e1266d8e678 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:31.831576 7022 raft_consensus.cc:2932] T 965b65ec896648698fbf3e1266d8e678 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:31.833673 4088 catalog_manager.cc:5484] T 965b65ec896648698fbf3e1266d8e678 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:31.835988 4364 raft_consensus.cc:2932] T 965b65ec896648698fbf3e1266d8e678 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:31.840272 7281 consensus_queue.cc:237] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } }
I20240501 20:10:31.849915 4498 raft_consensus.cc:1261] T 965b65ec896648698fbf3e1266d8e678 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:31.850557 4364 raft_consensus.cc:1261] T 965b65ec896648698fbf3e1266d8e678 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:31.850648 11311 consensus_queue.cc:1035] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:31.851974 11581 consensus_queue.cc:1035] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:31.855724 11581 raft_consensus.cc:2932] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:31.858378 7281 raft_consensus.cc:1261] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:31.859047 11050 consensus_queue.cc:1035] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:31.859983 4498 raft_consensus.cc:2932] T 965b65ec896648698fbf3e1266d8e678 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:31.860438 7420 raft_consensus.cc:1261] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:31.860541 4364 raft_consensus.cc:2932] T 965b65ec896648698fbf3e1266d8e678 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:31.863132 11408 consensus_queue.cc:1035] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:31.867810 4084 catalog_manager.cc:5484] T 965b65ec896648698fbf3e1266d8e678 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:31.872715 11118 raft_consensus.cc:1050] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e: attempting to promote NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d to VOTER
I20240501 20:10:31.873543 11118 consensus_queue.cc:237] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:31.873760 4077 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 965b65ec896648698fbf3e1266d8e678 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:31.879106 4364 raft_consensus.cc:1261] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:31.879591 11324 consensus_queue.cc:1035] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:31.881948 11050 raft_consensus.cc:2932] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:31.882586 4364 raft_consensus.cc:2932] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:31.887164 4364 raft_consensus.cc:1203] T 537b5ca846b5485ca1d98351d9038430 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:31.887256 7420 raft_consensus.cc:2932] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:31.888547 7001 tablet_service.cc:1514] Processing DeleteTablet for tablet 965b65ec896648698fbf3e1266d8e678 with delete_type TABLET_DATA_TOMBSTONED (TS ef0a4ada548446b3a531cb39a3c4e38d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33996
I20240501 20:10:31.889230 4088 catalog_manager.cc:5484] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index 7 to 8, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:31.896452 7022 raft_consensus.cc:1261] T 25f01a01106f47c489e252c928cf28bb P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:31.896831 7153 raft_consensus.cc:1261] T 25f01a01106f47c489e252c928cf28bb P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:31.897383 11118 consensus_queue.cc:1035] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:31.898017 11488 consensus_queue.cc:1035] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:31.902976 7287 raft_consensus.cc:1261] T 25f01a01106f47c489e252c928cf28bb P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:31.905634 11118 consensus_queue.cc:1035] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:31.937494 11600 ts_tablet_manager.cc:902] T 04a5fb2b808d466491144e64eb54d0fd P ef0a4ada548446b3a531cb39a3c4e38d: Initiating tablet copy from peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:31.941948 11603 ts_tablet_manager.cc:902] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd: Initiating tablet copy from peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:31.944165 11595 tablet_replica.cc:330] T 965b65ec896648698fbf3e1266d8e678 P ef0a4ada548446b3a531cb39a3c4e38d: stopping tablet replica
I20240501 20:10:31.944615 11595 raft_consensus.cc:2229] T 965b65ec896648698fbf3e1266d8e678 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:31.945048 11595 raft_consensus.cc:2258] T 965b65ec896648698fbf3e1266d8e678 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:31.948042 11488 raft_consensus.cc:2932] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:31.948685 7283 raft_consensus.cc:2932] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:31.949980 7153 raft_consensus.cc:2932] T 25f01a01106f47c489e252c928cf28bb P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:31.957876 4087 catalog_manager.cc:5484] T 25f01a01106f47c489e252c928cf28bb P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 2 to 3, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:31.959445 11595 ts_tablet_manager.cc:1875] T 965b65ec896648698fbf3e1266d8e678 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:31.963516 7283 raft_consensus.cc:1203] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:10:31.966262 11595 ts_tablet_manager.cc:1888] T 965b65ec896648698fbf3e1266d8e678 P ef0a4ada548446b3a531cb39a3c4e38d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:31.966508 11595 log.cc:1196] T 965b65ec896648698fbf3e1266d8e678 P ef0a4ada548446b3a531cb39a3c4e38d: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/wals/965b65ec896648698fbf3e1266d8e678
I20240501 20:10:31.967530 4078 catalog_manager.cc:4830] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): tablet 965b65ec896648698fbf3e1266d8e678 (table rebalancer_extra_table_22 [id=b43133d24f1646bca3476aba5ea45126]) successfully deleted
I20240501 20:10:31.954870 7022 raft_consensus.cc:2932] T 25f01a01106f47c489e252c928cf28bb P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:31.982987 7287 raft_consensus.cc:2932] T 25f01a01106f47c489e252c928cf28bb P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:31.997921 4628 consensus_queue.cc:237] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:32.000701 7287 raft_consensus.cc:1261] T 25f01a01106f47c489e252c928cf28bb P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:32.001338 7022 raft_consensus.cc:1261] T 25f01a01106f47c489e252c928cf28bb P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:32.001333 11118 consensus_queue.cc:1035] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:32.006816 11605 consensus_queue.cc:1035] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:32.007169 11610 raft_consensus.cc:2932] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:32.009871 7287 raft_consensus.cc:2932] T 25f01a01106f47c489e252c928cf28bb P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:32.013985 4079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 25f01a01106f47c489e252c928cf28bb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:32.015026 7022 raft_consensus.cc:2932] T 25f01a01106f47c489e252c928cf28bb P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:32.020390 4087 catalog_manager.cc:5484] T 25f01a01106f47c489e252c928cf28bb P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:32.028465 7135 tablet_service.cc:1514] Processing DeleteTablet for tablet 25f01a01106f47c489e252c928cf28bb with delete_type TABLET_DATA_TOMBSTONED (TS db0742ee3860468590eefef6ae6074fd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52036
I20240501 20:10:32.031200 11619 tablet_replica.cc:330] T 25f01a01106f47c489e252c928cf28bb P db0742ee3860468590eefef6ae6074fd: stopping tablet replica
I20240501 20:10:32.032112 11251 raft_consensus.cc:1050] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a: attempting to promote NON_VOTER dca57823003e4729b769ca12367d7530 to VOTER
I20240501 20:10:32.033053 11251 consensus_queue.cc:237] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:32.040577 7287 raft_consensus.cc:1261] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:32.043999 11528 consensus_queue.cc:1035] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:32.045002 4234 raft_consensus.cc:1261] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:32.046226 11528 consensus_queue.cc:1035] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:32.049850 11600 tablet_copy_client.cc:308] T 04a5fb2b808d466491144e64eb54d0fd P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.68:45205
I20240501 20:10:32.050974 4651 tablet_copy_service.cc:140] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Received BeginTabletCopySession request for tablet 04a5fb2b808d466491144e64eb54d0fd from peer ef0a4ada548446b3a531cb39a3c4e38d ({username='slave'} at 127.29.219.69:59565)
I20240501 20:10:32.051121 4651 tablet_copy_service.cc:161] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Beginning new tablet copy session on tablet 04a5fb2b808d466491144e64eb54d0fd from peer ef0a4ada548446b3a531cb39a3c4e38d at {username='slave'} at 127.29.219.69:59565: session id = ef0a4ada548446b3a531cb39a3c4e38d-04a5fb2b808d466491144e64eb54d0fd
I20240501 20:10:32.053644 4651 tablet_copy_source_session.cc:205] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:32.054960 11600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a5fb2b808d466491144e64eb54d0fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:32.056468 11528 raft_consensus.cc:2932] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:32.060120 4234 raft_consensus.cc:2932] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:32.062286 11600 tablet_copy_client.cc:783] T 04a5fb2b808d466491144e64eb54d0fd P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:32.062669 11600 tablet_copy_client.cc:647] T 04a5fb2b808d466491144e64eb54d0fd P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:32.064505 11600 tablet_copy_client.cc:523] T 04a5fb2b808d466491144e64eb54d0fd P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:32.067629 4498 raft_consensus.cc:1261] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:32.069479 11600 tablet_bootstrap.cc:492] T 04a5fb2b808d466491144e64eb54d0fd P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:32.071501 4087 catalog_manager.cc:5484] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 7 to 8, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:32.072154 11614 consensus_queue.cc:1035] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:32.071911 7287 raft_consensus.cc:2932] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:32.077782 4365 consensus_queue.cc:237] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:32.051110 11603 tablet_copy_client.cc:308] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.65:42941
I20240501 20:10:32.078971 11619 raft_consensus.cc:2229] T 25f01a01106f47c489e252c928cf28bb P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:32.079291 11619 raft_consensus.cc:2258] T 25f01a01106f47c489e252c928cf28bb P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:32.079375 4254 tablet_copy_service.cc:140] P dca57823003e4729b769ca12367d7530: Received BeginTabletCopySession request for tablet d690199665174c7d9bdbd54d2af4ac70 from peer db0742ee3860468590eefef6ae6074fd ({username='slave'} at 127.29.219.70:47215)
I20240501 20:10:32.079517 4254 tablet_copy_service.cc:161] P dca57823003e4729b769ca12367d7530: Beginning new tablet copy session on tablet d690199665174c7d9bdbd54d2af4ac70 from peer db0742ee3860468590eefef6ae6074fd at {username='slave'} at 127.29.219.70:47215: session id = db0742ee3860468590eefef6ae6074fd-d690199665174c7d9bdbd54d2af4ac70
I20240501 20:10:32.080556 11619 ts_tablet_manager.cc:1875] T 25f01a01106f47c489e252c928cf28bb P db0742ee3860468590eefef6ae6074fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:32.082401 4254 tablet_copy_source_session.cc:205] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:32.084183 11603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d690199665174c7d9bdbd54d2af4ac70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:32.087626 4232 raft_consensus.cc:1261] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:32.088676 4498 raft_consensus.cc:2932] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:32.090056 7281 raft_consensus.cc:1261] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:32.090904 11528 consensus_queue.cc:1035] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:32.091569 11251 consensus_queue.cc:1035] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:32.095400 11619 ts_tablet_manager.cc:1888] T 25f01a01106f47c489e252c928cf28bb P db0742ee3860468590eefef6ae6074fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:32.095065 11614 raft_consensus.cc:2932] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:32.095688 11619 log.cc:1196] T 25f01a01106f47c489e252c928cf28bb P db0742ee3860468590eefef6ae6074fd: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/wals/25f01a01106f47c489e252c928cf28bb
I20240501 20:10:32.095997 7281 raft_consensus.cc:2932] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:32.098927 4076 catalog_manager.cc:4830] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): tablet 25f01a01106f47c489e252c928cf28bb (table rebalancer_extra_table_14 [id=be6eb76edc624f598b8543519c4d4476]) successfully deleted
I20240501 20:10:32.099320 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f40b062e103d4e54a913fa0f4111f623 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:32.106688 11603 tablet_copy_client.cc:783] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:32.107524 11603 tablet_copy_client.cc:647] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:32.110087 11600 tablet_bootstrap.cc:492] T 04a5fb2b808d466491144e64eb54d0fd P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:32.109828 4087 catalog_manager.cc:5484] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index 8 to 9, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:32.110332 11603 tablet_copy_client.cc:523] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:32.114249 11600 tablet_bootstrap.cc:492] T 04a5fb2b808d466491144e64eb54d0fd P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap complete.
I20240501 20:10:32.114367 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet f40b062e103d4e54a913fa0f4111f623 with delete_type TABLET_DATA_TOMBSTONED (TS 3e974c03d3c546f9ab6c91e762a5426b not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:32.114985 11603 tablet_bootstrap.cc:492] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:32.115198 11600 ts_tablet_manager.cc:1367] T 04a5fb2b808d466491144e64eb54d0fd P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.046s user 0.010s sys 0.001s
I20240501 20:10:32.117260 11600 raft_consensus.cc:348] T 04a5fb2b808d466491144e64eb54d0fd P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:32.117543 11600 raft_consensus.cc:729] T 04a5fb2b808d466491144e64eb54d0fd P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: LEARNER
I20240501 20:10:32.117776 11600 consensus_queue.cc:260] T 04a5fb2b808d466491144e64eb54d0fd P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:32.119338 4234 raft_consensus.cc:2932] T f40b062e103d4e54a913fa0f4111f623 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:32.126978 11600 ts_tablet_manager.cc:1398] T 04a5fb2b808d466491144e64eb54d0fd P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 20:10:32.128362 4651 tablet_copy_service.cc:342] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Request end of tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-04a5fb2b808d466491144e64eb54d0fd received from {username='slave'} at 127.29.219.69:59565
I20240501 20:10:32.128546 4651 tablet_copy_service.cc:433] P 1d34e18a1d394d3f91c9f50d1c7bea4e: ending tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-04a5fb2b808d466491144e64eb54d0fd on tablet 04a5fb2b808d466491144e64eb54d0fd with peer ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:32.138950 11632 tablet_replica.cc:330] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:32.139245 11632 raft_consensus.cc:2229] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:32.139462 11632 raft_consensus.cc:2258] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:32.140426 11632 ts_tablet_manager.cc:1875] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:32.147313 11632 ts_tablet_manager.cc:1888] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:10:32.147502 11632 log.cc:1196] T f40b062e103d4e54a913fa0f4111f623 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/f40b062e103d4e54a913fa0f4111f623
I20240501 20:10:32.148380 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet f40b062e103d4e54a913fa0f4111f623 (table rebalance_test_table_7 [id=59dd8401cbd048cea30daf86baac8cfa]) successfully deleted
I20240501 20:10:32.195081 11603 tablet_bootstrap.cc:492] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:32.195600 11603 tablet_bootstrap.cc:492] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd: Bootstrap complete.
I20240501 20:10:32.195976 11603 ts_tablet_manager.cc:1367] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.081s user 0.049s sys 0.001s
I20240501 20:10:32.196749 11603 raft_consensus.cc:348] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:32.197028 11603 raft_consensus.cc:729] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: LEARNER
I20240501 20:10:32.197256 11603 consensus_queue.cc:260] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:32.199120 11603 ts_tablet_manager.cc:1398] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:32.199851 4254 tablet_copy_service.cc:342] P dca57823003e4729b769ca12367d7530: Request end of tablet copy session db0742ee3860468590eefef6ae6074fd-d690199665174c7d9bdbd54d2af4ac70 received from {username='slave'} at 127.29.219.70:47215
I20240501 20:10:32.199991 4254 tablet_copy_service.cc:433] P dca57823003e4729b769ca12367d7530: ending tablet copy session db0742ee3860468590eefef6ae6074fd-d690199665174c7d9bdbd54d2af4ac70 on tablet d690199665174c7d9bdbd54d2af4ac70 with peer db0742ee3860468590eefef6ae6074fd
I20240501 20:10:32.243193 4498 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ffaf0a0105ea4e1f9ed9d90c221ebc8a"
dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54642
I20240501 20:10:32.243438 4498 raft_consensus.cc:595] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Received request to transfer leadership
I20240501 20:10:32.280714 11050 raft_consensus.cc:979] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b: : Instructing follower 10b47782558c41ee91f5e164a372049a to start an election
I20240501 20:10:32.280869 11050 raft_consensus.cc:1067] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Signalling peer 10b47782558c41ee91f5e164a372049a to start an election
I20240501 20:10:32.281615 4365 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ffaf0a0105ea4e1f9ed9d90c221ebc8a"
dest_uuid: "10b47782558c41ee91f5e164a372049a"
from {username='slave'} at 127.29.219.67:52009
I20240501 20:10:32.281916 4365 raft_consensus.cc:482] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:10:32.282078 4365 raft_consensus.cc:3037] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:32.284618 4365 raft_consensus.cc:504] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:32.285796 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffaf0a0105ea4e1f9ed9d90c221ebc8a" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:10:32.285854 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffaf0a0105ea4e1f9ed9d90c221ebc8a" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:32.286010 4498 raft_consensus.cc:3032] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:10:32.286070 4498 raft_consensus.cc:729] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:10:32.286085 7420 raft_consensus.cc:3037] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:32.286307 4498 consensus_queue.cc:260] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:32.286723 4498 raft_consensus.cc:3037] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:32.288821 4365 leader_election.cc:288] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 2 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:32.289371 4498 raft_consensus.cc:2454] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 2.
I20240501 20:10:32.289986 7281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffaf0a0105ea4e1f9ed9d90c221ebc8a" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:32.290179 7420 raft_consensus.cc:2454] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 2.
I20240501 20:10:32.290225 7281 raft_consensus.cc:3037] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:32.290959 4301 leader_election.cc:302] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b, b9957d4086bc4a76b18ed9265a5739d5; no voters:
I20240501 20:10:32.291203 11528 raft_consensus.cc:2781] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:32.291322 11528 raft_consensus.cc:686] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [term 2 LEADER]: Becoming Leader. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Running, Role: LEADER
I20240501 20:10:32.291566 11528 consensus_queue.cc:237] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:32.293180 7281 raft_consensus.cc:2454] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 2.
I20240501 20:10:32.295650 4088 catalog_manager.cc:5484] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a reported cstate change: term changed from 1 to 2, leader changed from 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) to 10b47782558c41ee91f5e164a372049a (127.29.219.66). New cstate: current_term: 2 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:32.423395 7022 raft_consensus.cc:1203] T 04a5fb2b808d466491144e64eb54d0fd P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:32.472868 7153 raft_consensus.cc:1203] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 20:10:32.482985 11101 raft_consensus.cc:1050] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5: attempting to promote NON_VOTER 10b47782558c41ee91f5e164a372049a to VOTER
I20240501 20:10:32.483713 11101 consensus_queue.cc:237] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } }
I20240501 20:10:32.486287 4365 raft_consensus.cc:1261] T 537b5ca846b5485ca1d98351d9038430 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:32.486886 7153 raft_consensus.cc:1261] T 537b5ca846b5485ca1d98351d9038430 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:32.486994 4628 raft_consensus.cc:1261] T 537b5ca846b5485ca1d98351d9038430 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:32.487511 11101 consensus_queue.cc:1035] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:32.487969 11077 consensus_queue.cc:1035] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:32.488535 11077 consensus_queue.cc:1035] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:32.491264 11101 raft_consensus.cc:2932] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:32.493531 4628 raft_consensus.cc:2932] T 537b5ca846b5485ca1d98351d9038430 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:32.497258 7153 raft_consensus.cc:2932] T 537b5ca846b5485ca1d98351d9038430 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:32.498401 4088 catalog_manager.cc:5484] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:32.506471 4365 raft_consensus.cc:2932] T 537b5ca846b5485ca1d98351d9038430 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:32.509095 7420 consensus_queue.cc:237] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } }
I20240501 20:10:32.511832 11050 raft_consensus.cc:1050] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b: attempting to promote NON_VOTER bceeae1c79964acabdae8ff771bf0c97 to VOTER
I20240501 20:10:32.512640 11050 consensus_queue.cc:237] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:32.512817 4365 raft_consensus.cc:1261] T 537b5ca846b5485ca1d98351d9038430 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:32.513393 11077 consensus_queue.cc:1035] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:32.513837 4628 raft_consensus.cc:1261] T 537b5ca846b5485ca1d98351d9038430 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:32.514354 11077 consensus_queue.cc:1035] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:32.514720 7281 raft_consensus.cc:1261] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:32.515424 4631 raft_consensus.cc:1261] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:32.515538 11324 consensus_queue.cc:1035] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:32.515874 11050 consensus_queue.cc:1035] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:32.516327 7022 raft_consensus.cc:1261] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:32.516880 11050 consensus_queue.cc:1035] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:32.519583 11077 raft_consensus.cc:2932] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:32.523056 4079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 537b5ca846b5485ca1d98351d9038430 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:32.523187 11050 raft_consensus.cc:2932] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:32.523522 4627 raft_consensus.cc:2932] T 537b5ca846b5485ca1d98351d9038430 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:32.528380 7281 raft_consensus.cc:2932] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:32.528890 7022 raft_consensus.cc:2932] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:32.530345 4631 raft_consensus.cc:2932] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:32.536389 4084 catalog_manager.cc:5484] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:32.538650 4089 catalog_manager.cc:5484] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: config changed from index 7 to 8, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:32.543388 7135 tablet_service.cc:1514] Processing DeleteTablet for tablet 537b5ca846b5485ca1d98351d9038430 with delete_type TABLET_DATA_TOMBSTONED (TS db0742ee3860468590eefef6ae6074fd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52036
I20240501 20:10:32.544498 4365 raft_consensus.cc:2932] T 537b5ca846b5485ca1d98351d9038430 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:32.545243 11619 tablet_replica.cc:330] T 537b5ca846b5485ca1d98351d9038430 P db0742ee3860468590eefef6ae6074fd: stopping tablet replica
I20240501 20:10:32.545442 11619 raft_consensus.cc:2229] T 537b5ca846b5485ca1d98351d9038430 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:32.545648 11619 raft_consensus.cc:2258] T 537b5ca846b5485ca1d98351d9038430 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:32.546718 11619 ts_tablet_manager.cc:1875] T 537b5ca846b5485ca1d98351d9038430 P db0742ee3860468590eefef6ae6074fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:32.555888 11619 ts_tablet_manager.cc:1888] T 537b5ca846b5485ca1d98351d9038430 P db0742ee3860468590eefef6ae6074fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:32.556046 11619 log.cc:1196] T 537b5ca846b5485ca1d98351d9038430 P db0742ee3860468590eefef6ae6074fd: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/wals/537b5ca846b5485ca1d98351d9038430
I20240501 20:10:32.558805 4076 catalog_manager.cc:4830] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): tablet 537b5ca846b5485ca1d98351d9038430 (table rebalancer_extra_table_10 [id=abd146b01560429fae00683c20d75e9c]) successfully deleted
I20240501 20:10:32.590564 4498 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "46df0faa681b469abe9e383b181a6053"
dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54682
I20240501 20:10:32.590791 4498 raft_consensus.cc:595] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Received request to transfer leadership
I20240501 20:10:32.676854 7416 raft_consensus.cc:1261] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:10:32.678097 11251 consensus_queue.cc:1035] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:32.701988 4498 raft_consensus.cc:1261] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:10:32.707494 11251 consensus_queue.cc:1035] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:32.720883 7281 raft_consensus.cc:1261] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:10:32.721894 11528 consensus_queue.cc:1035] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:32.784847 4496 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "46df0faa681b469abe9e383b181a6053"
dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54698
I20240501 20:10:32.785060 4496 raft_consensus.cc:595] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Received request to transfer leadership
I20240501 20:10:32.929241 11050 raft_consensus.cc:979] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b: : Instructing follower bceeae1c79964acabdae8ff771bf0c97 to start an election
I20240501 20:10:32.929370 11050 raft_consensus.cc:1067] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Signalling peer bceeae1c79964acabdae8ff771bf0c97 to start an election
I20240501 20:10:32.929981 7287 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46df0faa681b469abe9e383b181a6053"
dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
from {username='slave'} at 127.29.219.67:40409
I20240501 20:10:32.930195 7287 raft_consensus.cc:482] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:10:32.930335 7287 raft_consensus.cc:3037] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:32.932446 7287 raft_consensus.cc:504] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:32.933199 7287 leader_election.cc:288] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 election: Requested vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:32.933516 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46df0faa681b469abe9e383b181a6053" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e"
I20240501 20:10:32.933733 4631 raft_consensus.cc:3037] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:32.933867 7019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46df0faa681b469abe9e383b181a6053" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "ef0a4ada548446b3a531cb39a3c4e38d"
I20240501 20:10:32.934068 7019 raft_consensus.cc:3037] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:32.936152 4631 raft_consensus.cc:2454] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 2.
I20240501 20:10:32.936404 7019 raft_consensus.cc:2454] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 2.
I20240501 20:10:32.936584 4496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46df0faa681b469abe9e383b181a6053" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:10:32.936806 4496 raft_consensus.cc:3032] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:10:32.936878 4496 raft_consensus.cc:729] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:10:32.936872 7222 leader_election.cc:302] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d; no voters:
I20240501 20:10:32.937211 11581 raft_consensus.cc:2781] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:32.937141 4496 consensus_queue.cc:260] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:32.937497 4496 raft_consensus.cc:3037] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:32.937711 11581 raft_consensus.cc:686] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97 [term 2 LEADER]: Becoming Leader. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Running, Role: LEADER
I20240501 20:10:32.938004 11581 consensus_queue.cc:237] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:32.939703 4496 raft_consensus.cc:2454] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 2.
I20240501 20:10:32.941956 4090 catalog_manager.cc:5484] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: term changed from 1 to 2, leader changed from 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) to bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71). New cstate: current_term: 2 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:32.984603 11687 raft_consensus.cc:1050] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530: attempting to promote NON_VOTER db0742ee3860468590eefef6ae6074fd to VOTER
I20240501 20:10:32.985342 11687 consensus_queue.cc:237] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:32.987928 7153 raft_consensus.cc:1261] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:32.988492 4496 raft_consensus.cc:1261] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:32.988631 11762 consensus_queue.cc:1035] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:32.989074 7416 raft_consensus.cc:1261] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:10:32.989066 11687 consensus_queue.cc:1035] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:32.990319 11146 consensus_queue.cc:1035] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:32.994077 11146 raft_consensus.cc:2932] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:32.994766 4496 raft_consensus.cc:2932] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:33.001384 4090 catalog_manager.cc:5484] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 7 to 8, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:33.002264 11610 raft_consensus.cc:1050] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e: attempting to promote NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d to VOTER
I20240501 20:10:33.003139 11610 consensus_queue.cc:237] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:33.004703 7416 raft_consensus.cc:2932] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:33.023339 7019 raft_consensus.cc:1261] T 04a5fb2b808d466491144e64eb54d0fd P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:33.024122 4364 raft_consensus.cc:1261] T 04a5fb2b808d466491144e64eb54d0fd P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:33.026275 11771 consensus_queue.cc:1035] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:33.026468 7152 raft_consensus.cc:1261] T 04a5fb2b808d466491144e64eb54d0fd P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:33.027187 11768 consensus_queue.cc:1035] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:33.033483 7154 raft_consensus.cc:2932] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:33.040143 4234 consensus_queue.cc:237] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:33.043614 7416 raft_consensus.cc:1261] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:33.044571 11146 consensus_queue.cc:1035] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:33.047237 11771 consensus_queue.cc:1035] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:33.047906 7155 raft_consensus.cc:1261] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:10:33.047549 11146 raft_consensus.cc:2932] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:33.051564 7416 raft_consensus.cc:2932] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:33.054041 11687 consensus_queue.cc:1035] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:33.055078 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d690199665174c7d9bdbd54d2af4ac70 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:33.058071 4090 catalog_manager.cc:5484] T d690199665174c7d9bdbd54d2af4ac70 P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 8 to 9, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:33.062578 7153 raft_consensus.cc:2932] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:33.065894 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet d690199665174c7d9bdbd54d2af4ac70 with delete_type TABLET_DATA_TOMBSTONED (TS 3e974c03d3c546f9ab6c91e762a5426b not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:33.067869 11771 raft_consensus.cc:2932] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:33.075882 7152 raft_consensus.cc:2932] T 04a5fb2b808d466491144e64eb54d0fd P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:33.078482 4087 catalog_manager.cc:5484] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 2 to 3, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:33.078292 4364 raft_consensus.cc:2932] T 04a5fb2b808d466491144e64eb54d0fd P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:33.119504 11779 tablet_replica.cc:330] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:33.119946 11779 raft_consensus.cc:2229] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:33.120339 11779 raft_consensus.cc:2258] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:33.121941 11779 ts_tablet_manager.cc:1875] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:33.125926 7019 raft_consensus.cc:2932] T 04a5fb2b808d466491144e64eb54d0fd P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:33.132586 11779 ts_tablet_manager.cc:1888] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 20:10:33.132778 11779 log.cc:1196] T d690199665174c7d9bdbd54d2af4ac70 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/d690199665174c7d9bdbd54d2af4ac70
I20240501 20:10:33.132802 4631 consensus_queue.cc:237] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:33.133795 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet d690199665174c7d9bdbd54d2af4ac70 (table rebalance_test_table_3 [id=5fb9ba92237649f89c99e4487cd45f29]) successfully deleted
I20240501 20:10:33.135836 7020 raft_consensus.cc:1261] T 04a5fb2b808d466491144e64eb54d0fd P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:33.135905 4364 raft_consensus.cc:1261] T 04a5fb2b808d466491144e64eb54d0fd P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:33.136763 11768 consensus_queue.cc:1035] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:33.137377 11768 consensus_queue.cc:1035] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:33.140123 11783 raft_consensus.cc:2932] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:33.140916 7019 raft_consensus.cc:2932] T 04a5fb2b808d466491144e64eb54d0fd P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:33.145298 4079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 04a5fb2b808d466491144e64eb54d0fd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:33.149092 4364 raft_consensus.cc:2932] T 04a5fb2b808d466491144e64eb54d0fd P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:33.149698 4087 catalog_manager.cc:5484] T 04a5fb2b808d466491144e64eb54d0fd P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:33.164304 7135 tablet_service.cc:1514] Processing DeleteTablet for tablet 04a5fb2b808d466491144e64eb54d0fd with delete_type TABLET_DATA_TOMBSTONED (TS db0742ee3860468590eefef6ae6074fd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52036
I20240501 20:10:33.180898 11792 tablet_replica.cc:330] T 04a5fb2b808d466491144e64eb54d0fd P db0742ee3860468590eefef6ae6074fd: stopping tablet replica
I20240501 20:10:33.181169 11792 raft_consensus.cc:2229] T 04a5fb2b808d466491144e64eb54d0fd P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:33.181468 11792 raft_consensus.cc:2258] T 04a5fb2b808d466491144e64eb54d0fd P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:33.183229 11792 ts_tablet_manager.cc:1875] T 04a5fb2b808d466491144e64eb54d0fd P db0742ee3860468590eefef6ae6074fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:33.190282 11792 ts_tablet_manager.cc:1888] T 04a5fb2b808d466491144e64eb54d0fd P db0742ee3860468590eefef6ae6074fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:33.190407 11792 log.cc:1196] T 04a5fb2b808d466491144e64eb54d0fd P db0742ee3860468590eefef6ae6074fd: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/wals/04a5fb2b808d466491144e64eb54d0fd
I20240501 20:10:33.191430 4076 catalog_manager.cc:4830] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): tablet 04a5fb2b808d466491144e64eb54d0fd (table rebalancer_extra_table_18 [id=4b4b9eb576884393874a7d89667da324]) successfully deleted
I20240501 20:10:33.230015 4364 consensus_queue.cc:237] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:33.232445 7419 raft_consensus.cc:1261] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:10:33.232827 7281 raft_consensus.cc:1261] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:10:33.233429 11743 consensus_queue.cc:1035] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:10:33.233973 11743 consensus_queue.cc:1035] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:10:33.237424 11743 raft_consensus.cc:2932] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 10b47782558c41ee91f5e164a372049a [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:33.237977 7419 raft_consensus.cc:2932] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:33.242480 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ffaf0a0105ea4e1f9ed9d90c221ebc8a with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:33.243319 7281 raft_consensus.cc:2932] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:33.245796 4087 catalog_manager.cc:5484] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: config changed from index 8 to 10, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New cstate: current_term: 2 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:33.252549 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet ffaf0a0105ea4e1f9ed9d90c221ebc8a with delete_type TABLET_DATA_TOMBSTONED (TS 3e974c03d3c546f9ab6c91e762a5426b not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:33.252900 11779 tablet_replica.cc:330] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:33.253094 11779 raft_consensus.cc:2229] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:33.253258 11779 raft_consensus.cc:2258] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:33.254351 11779 ts_tablet_manager.cc:1875] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:33.260923 11779 ts_tablet_manager.cc:1888] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 20:10:33.261039 11779 log.cc:1196] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/ffaf0a0105ea4e1f9ed9d90c221ebc8a
I20240501 20:10:33.261765 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet ffaf0a0105ea4e1f9ed9d90c221ebc8a (table rebalance_test_table_17 [id=09dd73eba68c4f8499a786750d39c1cf]) successfully deleted
I20240501 20:10:33.438722 4498 raft_consensus.cc:1261] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:10:33.439357 11581 consensus_queue.cc:1035] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:33.461369 7019 raft_consensus.cc:1261] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d [term 2 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:10:33.461982 11581 consensus_queue.cc:1035] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:33.465504 4627 raft_consensus.cc:1261] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 20:10:33.466434 11581 consensus_queue.cc:1035] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:10:34.087613 7287 consensus_queue.cc:237] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:34.089498 7019 raft_consensus.cc:1261] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d [term 2 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:10:34.089587 4627 raft_consensus.cc:1261] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 20:10:34.090140 11804 consensus_queue.cc:1035] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:10:34.090476 11581 consensus_queue.cc:1035] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:10:34.092792 11828 raft_consensus.cc:2932] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:34.093274 7020 raft_consensus.cc:2932] T 46df0faa681b469abe9e383b181a6053 P ef0a4ada548446b3a531cb39a3c4e38d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:34.093272 4627 raft_consensus.cc:2932] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:34.098189 4077 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 46df0faa681b469abe9e383b181a6053 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:34.100425 4090 catalog_manager.cc:5484] T 46df0faa681b469abe9e383b181a6053 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 8 to 10, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New cstate: current_term: 2 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:34.106575 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet 46df0faa681b469abe9e383b181a6053 with delete_type TABLET_DATA_TOMBSTONED (TS 3e974c03d3c546f9ab6c91e762a5426b not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:34.115025 11833 tablet_replica.cc:330] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:34.115252 11833 raft_consensus.cc:2229] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:34.115445 11833 raft_consensus.cc:2258] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:34.116371 11833 ts_tablet_manager.cc:1875] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:34.122467 11833 ts_tablet_manager.cc:1888] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240501 20:10:34.122673 11833 log.cc:1196] T 46df0faa681b469abe9e383b181a6053 P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/46df0faa681b469abe9e383b181a6053
I20240501 20:10:34.123663 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet 46df0faa681b469abe9e383b181a6053 (table rebalance_test_table_13 [id=e21b0047cbf842c6833a78db81a34171]) successfully deleted
I20240501 20:10:34.478364 4346 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:34.482813 4610 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 20:10:34.522037 4478 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:34.528232 7400 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 20:10:34.534221 7267 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:34.550889 7135 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:34.616201 4215 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 20:10:34.633345 7001 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:10:34.846647 4234 consensus_queue.cc:237] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:34.849182 4364 raft_consensus.cc:1261] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:34.849460 4631 raft_consensus.cc:1261] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:34.849926 4170 consensus_peers.cc:485] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 -> Peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): Couldn't send request to peer b9957d4086bc4a76b18ed9265a5739d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d3bb8370e7c41dc9cf7f1c835ac275b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:34.849978 11146 consensus_queue.cc:1035] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:34.850349 11897 consensus_queue.cc:1035] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:34.853364 11146 raft_consensus.cc:2932] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:34.854117 4627 raft_consensus.cc:2932] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:34.857745 4364 raft_consensus.cc:2932] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:34.861449 4091 catalog_manager.cc:5484] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index -1 to 2, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:34.880623 7153 consensus_queue.cc:237] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:34.884521 4234 raft_consensus.cc:1261] T 024c925714cb419aaa8870a053a6ca2f P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:34.884626 7287 raft_consensus.cc:1261] T 024c925714cb419aaa8870a053a6ca2f P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:34.884675 7090 consensus_peers.cc:485] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 024c925714cb419aaa8870a053a6ca2f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:34.885144 11816 consensus_queue.cc:1035] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:34.885433 11016 consensus_queue.cc:1035] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:34.888777 11816 raft_consensus.cc:2932] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:34.897099 4234 raft_consensus.cc:2932] T 024c925714cb419aaa8870a053a6ca2f P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:34.908210 4093 catalog_manager.cc:5484] T 024c925714cb419aaa8870a053a6ca2f P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index -1 to 2, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:34.908684 7281 raft_consensus.cc:2932] T 024c925714cb419aaa8870a053a6ca2f P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:34.929194 7281 consensus_queue.cc:237] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:34.931664 7416 raft_consensus.cc:1261] T 376ae6034f394a0fb0cab0d8a9241134 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:34.931815 4234 raft_consensus.cc:1261] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:34.932161 7222 consensus_peers.cc:485] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 376ae6034f394a0fb0cab0d8a9241134. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:34.932201 11930 consensus_queue.cc:1035] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:34.932534 11804 consensus_queue.cc:1035] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:34.935444 11804 raft_consensus.cc:2932] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:34.939328 7416 raft_consensus.cc:2932] T 376ae6034f394a0fb0cab0d8a9241134 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:34.942751 4093 catalog_manager.cc:5484] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:34.958233 4234 raft_consensus.cc:2932] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:34.975255 4234 consensus_queue.cc:237] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:34.986878 7416 raft_consensus.cc:1261] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:34.986984 4167 consensus_peers.cc:485] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 -> Peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Couldn't send request to peer db0742ee3860468590eefef6ae6074fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82260307259e4a9fb12231ff4abd79c5. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:34.987017 4364 raft_consensus.cc:1261] T 82260307259e4a9fb12231ff4abd79c5 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:34.987702 11910 consensus_queue.cc:1035] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:34.988749 11910 consensus_queue.cc:1035] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:34.990304 11146 raft_consensus.cc:2932] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:34.992480 7416 raft_consensus.cc:2932] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:34.993958 4364 raft_consensus.cc:2932] T 82260307259e4a9fb12231ff4abd79c5 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:34.997447 4093 catalog_manager.cc:5484] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index -1 to 2, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:35.014334 4498 consensus_queue.cc:237] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:35.016862 7416 raft_consensus.cc:1261] T 786296114e8a47c68df103934ec055d5 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:35.017082 7281 raft_consensus.cc:1261] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:35.017598 11924 consensus_queue.cc:1035] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:35.017839 4435 consensus_peers.cc:485] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b -> Peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Couldn't send request to peer 1d34e18a1d394d3f91c9f50d1c7bea4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 786296114e8a47c68df103934ec055d5. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:35.018085 11050 consensus_queue.cc:1035] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:35.020992 11924 raft_consensus.cc:2932] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:35.021715 7416 raft_consensus.cc:2932] T 786296114e8a47c68df103934ec055d5 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:35.023278 7287 raft_consensus.cc:2932] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:35.029003 4091 catalog_manager.cc:5484] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:35.044561 7153 consensus_queue.cc:237] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
W20240501 20:10:35.047247 7090 consensus_peers.cc:485] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92ce2c1ce3e0449bab5490b690721cbf. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:35.048003 7416 raft_consensus.cc:1261] T 92ce2c1ce3e0449bab5490b690721cbf P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:35.048334 7287 raft_consensus.cc:1261] T 92ce2c1ce3e0449bab5490b690721cbf P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:35.048607 11816 consensus_queue.cc:1035] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:35.049118 11920 consensus_queue.cc:1035] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:35.051916 11816 raft_consensus.cc:2932] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:35.052538 7281 raft_consensus.cc:2932] T 92ce2c1ce3e0449bab5490b690721cbf P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:35.059422 4089 catalog_manager.cc:5484] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:35.060415 7416 raft_consensus.cc:2932] T 92ce2c1ce3e0449bab5490b690721cbf P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:35.086457 7019 consensus_queue.cc:237] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:35.088735 7153 raft_consensus.cc:1261] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:35.089198 4364 raft_consensus.cc:1261] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:35.090204 11836 consensus_queue.cc:1035] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:35.090759 6957 consensus_peers.cc:485] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd21a0faa3bb487cb80f0951b12b3d67. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:35.091203 11840 consensus_queue.cc:1035] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:35.093930 11840 raft_consensus.cc:2932] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:35.095275 4364 raft_consensus.cc:2932] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:35.098564 7153 raft_consensus.cc:2932] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:35.103613 4089 catalog_manager.cc:5484] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:35.132328 7281 consensus_queue.cc:237] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:35.135071 4364 raft_consensus.cc:1261] T ce0b97a1badb4935a4553902e6feb20b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:35.135350 4498 raft_consensus.cc:1261] T ce0b97a1badb4935a4553902e6feb20b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:35.135668 11828 consensus_queue.cc:1035] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:35.136243 7220 consensus_peers.cc:485] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 -> Peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Couldn't send request to peer db0742ee3860468590eefef6ae6074fd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce0b97a1badb4935a4553902e6feb20b. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:35.137861 11828 consensus_queue.cc:1035] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:35.139788 11804 raft_consensus.cc:2932] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:35.145229 4364 raft_consensus.cc:2932] T ce0b97a1badb4935a4553902e6feb20b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:35.146939 4091 catalog_manager.cc:5484] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index -1 to 2, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:35.162308 4498 raft_consensus.cc:2932] T ce0b97a1badb4935a4553902e6feb20b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } } }
I20240501 20:10:35.190084 7153 consensus_queue.cc:237] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:35.192562 4627 raft_consensus.cc:1261] T 9709a8bb107742b79078f91da9fab8f6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:35.192597 7416 raft_consensus.cc:1261] T 9709a8bb107742b79078f91da9fab8f6 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:35.193099 11016 consensus_queue.cc:1035] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:35.193424 7090 consensus_peers.cc:485] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9709a8bb107742b79078f91da9fab8f6. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:35.193506 11920 consensus_queue.cc:1035] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:35.196362 11016 raft_consensus.cc:2932] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:35.206769 7419 raft_consensus.cc:2932] T 9709a8bb107742b79078f91da9fab8f6 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:35.217221 4631 raft_consensus.cc:2932] T 9709a8bb107742b79078f91da9fab8f6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:35.224546 4091 catalog_manager.cc:5484] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:35.263418 7419 consensus_queue.cc:237] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
W20240501 20:10:35.266471 7356 consensus_peers.cc:485] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Couldn't send request to peer 1d34e18a1d394d3f91c9f50d1c7bea4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e0ecc1043af4bc1ab31dfa80ea47bb4. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:35.266476 4498 raft_consensus.cc:1261] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:35.266520 7281 raft_consensus.cc:1261] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:35.266973 11077 consensus_queue.cc:1035] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:35.267277 11991 consensus_queue.cc:1035] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:35.269944 11077 raft_consensus.cc:2932] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:35.270747 4498 raft_consensus.cc:2932] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:35.271857 7281 raft_consensus.cc:2932] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:35.278291 4091 catalog_manager.cc:5484] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:35.318563 7019 consensus_queue.cc:237] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:35.322844 4496 raft_consensus.cc:1261] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:35.322865 7152 raft_consensus.cc:1261] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:35.323408 11836 consensus_queue.cc:1035] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:35.323714 6958 consensus_peers.cc:485] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d -> Peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): Couldn't send request to peer b9957d4086bc4a76b18ed9265a5739d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99bd0e9f4cd7496785ad71480d7d3d61. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:35.323850 11840 consensus_queue.cc:1035] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:35.326889 11836 raft_consensus.cc:2932] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:35.327257 4498 raft_consensus.cc:2932] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:35.328105 7152 raft_consensus.cc:2932] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } } }
I20240501 20:10:35.334115 4093 catalog_manager.cc:5484] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: config changed from index -1 to 2, NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) added. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:35.345693 12008 ts_tablet_manager.cc:902] T 024c925714cb419aaa8870a053a6ca2f P ef0a4ada548446b3a531cb39a3c4e38d: Initiating tablet copy from peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:35.355469 7419 consensus_queue.cc:237] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:35.356104 12008 tablet_copy_client.cc:308] T 024c925714cb419aaa8870a053a6ca2f P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.70:36321
I20240501 20:10:35.356784 7175 tablet_copy_service.cc:140] P db0742ee3860468590eefef6ae6074fd: Received BeginTabletCopySession request for tablet 024c925714cb419aaa8870a053a6ca2f from peer ef0a4ada548446b3a531cb39a3c4e38d ({username='slave'} at 127.29.219.69:39809)
I20240501 20:10:35.356946 7175 tablet_copy_service.cc:161] P db0742ee3860468590eefef6ae6074fd: Beginning new tablet copy session on tablet 024c925714cb419aaa8870a053a6ca2f from peer ef0a4ada548446b3a531cb39a3c4e38d at {username='slave'} at 127.29.219.69:39809: session id = ef0a4ada548446b3a531cb39a3c4e38d-024c925714cb419aaa8870a053a6ca2f
I20240501 20:10:35.358418 4498 raft_consensus.cc:1261] T 9c77905da1b3453fa9a83366f7c51c37 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:35.358886 7019 raft_consensus.cc:1261] T 9c77905da1b3453fa9a83366f7c51c37 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:35.359004 11991 consensus_queue.cc:1035] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:35.359378 7355 consensus_peers.cc:485] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9c77905da1b3453fa9a83366f7c51c37. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:35.359647 11991 consensus_queue.cc:1035] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:35.359807 7175 tablet_copy_source_session.cc:205] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:35.361392 12008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 024c925714cb419aaa8870a053a6ca2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:35.362437 11991 raft_consensus.cc:2932] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:35.366948 12008 tablet_copy_client.cc:783] T 024c925714cb419aaa8870a053a6ca2f P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:35.367225 12008 tablet_copy_client.cc:647] T 024c925714cb419aaa8870a053a6ca2f P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:35.367365 4498 raft_consensus.cc:2932] T 9c77905da1b3453fa9a83366f7c51c37 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:35.368703 12008 tablet_copy_client.cc:523] T 024c925714cb419aaa8870a053a6ca2f P ef0a4ada548446b3a531cb39a3c4e38d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:35.371825 12008 tablet_bootstrap.cc:492] T 024c925714cb419aaa8870a053a6ca2f P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap starting.
I20240501 20:10:35.377425 4093 catalog_manager.cc:5484] T 9c77905da1b3453fa9a83366f7c51c37 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:35.383946 12017 ts_tablet_manager.cc:902] T 1d3bb8370e7c41dc9cf7f1c835ac275b P b9957d4086bc4a76b18ed9265a5739d5: Initiating tablet copy from peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:35.389403 12008 tablet_bootstrap.cc:492] T 024c925714cb419aaa8870a053a6ca2f P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:35.389869 12008 tablet_bootstrap.cc:492] T 024c925714cb419aaa8870a053a6ca2f P ef0a4ada548446b3a531cb39a3c4e38d: Bootstrap complete.
I20240501 20:10:35.390117 12008 ts_tablet_manager.cc:1367] T 024c925714cb419aaa8870a053a6ca2f P ef0a4ada548446b3a531cb39a3c4e38d: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.004s
I20240501 20:10:35.390983 12008 raft_consensus.cc:348] T 024c925714cb419aaa8870a053a6ca2f P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:35.391235 12008 raft_consensus.cc:729] T 024c925714cb419aaa8870a053a6ca2f P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ef0a4ada548446b3a531cb39a3c4e38d, State: Initialized, Role: LEARNER
I20240501 20:10:35.391459 12008 consensus_queue.cc:260] T 024c925714cb419aaa8870a053a6ca2f P ef0a4ada548446b3a531cb39a3c4e38d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:35.393661 12008 ts_tablet_manager.cc:1398] T 024c925714cb419aaa8870a053a6ca2f P ef0a4ada548446b3a531cb39a3c4e38d: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240501 20:10:35.394229 7175 tablet_copy_service.cc:342] P db0742ee3860468590eefef6ae6074fd: Request end of tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-024c925714cb419aaa8870a053a6ca2f received from {username='slave'} at 127.29.219.69:39809
I20240501 20:10:35.394366 7175 tablet_copy_service.cc:433] P db0742ee3860468590eefef6ae6074fd: ending tablet copy session ef0a4ada548446b3a531cb39a3c4e38d-024c925714cb419aaa8870a053a6ca2f on tablet 024c925714cb419aaa8870a053a6ca2f with peer ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:35.411334 7019 raft_consensus.cc:2932] T 9c77905da1b3453fa9a83366f7c51c37 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } } }
I20240501 20:10:35.417486 12017 tablet_copy_client.cc:308] T 1d3bb8370e7c41dc9cf7f1c835ac275b P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.65:42941
I20240501 20:10:35.418968 4254 tablet_copy_service.cc:140] P dca57823003e4729b769ca12367d7530: Received BeginTabletCopySession request for tablet 1d3bb8370e7c41dc9cf7f1c835ac275b from peer b9957d4086bc4a76b18ed9265a5739d5 ({username='slave'} at 127.29.219.72:48035)
I20240501 20:10:35.419140 4254 tablet_copy_service.cc:161] P dca57823003e4729b769ca12367d7530: Beginning new tablet copy session on tablet 1d3bb8370e7c41dc9cf7f1c835ac275b from peer b9957d4086bc4a76b18ed9265a5739d5 at {username='slave'} at 127.29.219.72:48035: session id = b9957d4086bc4a76b18ed9265a5739d5-1d3bb8370e7c41dc9cf7f1c835ac275b
I20240501 20:10:35.421759 4254 tablet_copy_source_session.cc:205] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:35.423038 12017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3bb8370e7c41dc9cf7f1c835ac275b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:35.429286 12017 tablet_copy_client.cc:783] T 1d3bb8370e7c41dc9cf7f1c835ac275b P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:35.429591 12017 tablet_copy_client.cc:647] T 1d3bb8370e7c41dc9cf7f1c835ac275b P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:35.431100 7019 consensus_queue.cc:237] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:35.431308 12017 tablet_copy_client.cc:523] T 1d3bb8370e7c41dc9cf7f1c835ac275b P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:35.433418 4498 raft_consensus.cc:1261] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:35.433609 7152 raft_consensus.cc:1261] T 56b2082919b548bfb56b939373caf770 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:35.434231 11840 consensus_queue.cc:1035] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:35.434432 12017 tablet_bootstrap.cc:492] T 1d3bb8370e7c41dc9cf7f1c835ac275b P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:35.434907 11840 consensus_queue.cc:1035] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:35.435492 6958 consensus_peers.cc:485] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d -> Peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Couldn't send request to peer 1d34e18a1d394d3f91c9f50d1c7bea4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 56b2082919b548bfb56b939373caf770. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:35.439188 12030 ts_tablet_manager.cc:902] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd: Initiating tablet copy from peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:35.450608 11965 raft_consensus.cc:2932] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:35.451476 4499 raft_consensus.cc:2932] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:35.456552 7152 raft_consensus.cc:2932] T 56b2082919b548bfb56b939373caf770 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:35.462288 4091 catalog_manager.cc:5484] T 56b2082919b548bfb56b939373caf770 P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index -1 to 2, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:35.465504 12017 tablet_bootstrap.cc:492] T 1d3bb8370e7c41dc9cf7f1c835ac275b P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:35.466081 12017 tablet_bootstrap.cc:492] T 1d3bb8370e7c41dc9cf7f1c835ac275b P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap complete.
I20240501 20:10:35.466454 12017 ts_tablet_manager.cc:1367] T 1d3bb8370e7c41dc9cf7f1c835ac275b P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.012s
I20240501 20:10:35.467541 12017 raft_consensus.cc:348] T 1d3bb8370e7c41dc9cf7f1c835ac275b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:35.467798 12017 raft_consensus.cc:729] T 1d3bb8370e7c41dc9cf7f1c835ac275b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: LEARNER
I20240501 20:10:35.468067 12017 consensus_queue.cc:260] T 1d3bb8370e7c41dc9cf7f1c835ac275b P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:35.468693 12017 ts_tablet_manager.cc:1398] T 1d3bb8370e7c41dc9cf7f1c835ac275b P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:35.469372 4254 tablet_copy_service.cc:342] P dca57823003e4729b769ca12367d7530: Request end of tablet copy session b9957d4086bc4a76b18ed9265a5739d5-1d3bb8370e7c41dc9cf7f1c835ac275b received from {username='slave'} at 127.29.219.72:48035
I20240501 20:10:35.469480 4254 tablet_copy_service.cc:433] P dca57823003e4729b769ca12367d7530: ending tablet copy session b9957d4086bc4a76b18ed9265a5739d5-1d3bb8370e7c41dc9cf7f1c835ac275b on tablet 1d3bb8370e7c41dc9cf7f1c835ac275b with peer b9957d4086bc4a76b18ed9265a5739d5
I20240501 20:10:35.477864 12038 ts_tablet_manager.cc:902] T 786296114e8a47c68df103934ec055d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Initiating tablet copy from peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:35.479665 12030 tablet_copy_client.cc:308] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.65:42941
I20240501 20:10:35.480387 4254 tablet_copy_service.cc:140] P dca57823003e4729b769ca12367d7530: Received BeginTabletCopySession request for tablet 82260307259e4a9fb12231ff4abd79c5 from peer db0742ee3860468590eefef6ae6074fd ({username='slave'} at 127.29.219.70:47215)
I20240501 20:10:35.480546 4254 tablet_copy_service.cc:161] P dca57823003e4729b769ca12367d7530: Beginning new tablet copy session on tablet 82260307259e4a9fb12231ff4abd79c5 from peer db0742ee3860468590eefef6ae6074fd at {username='slave'} at 127.29.219.70:47215: session id = db0742ee3860468590eefef6ae6074fd-82260307259e4a9fb12231ff4abd79c5
I20240501 20:10:35.483328 4254 tablet_copy_source_session.cc:205] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:35.483444 12038 tablet_copy_client.cc:308] T 786296114e8a47c68df103934ec055d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.67:34595
I20240501 20:10:35.484066 4519 tablet_copy_service.cc:140] P 3e974c03d3c546f9ab6c91e762a5426b: Received BeginTabletCopySession request for tablet 786296114e8a47c68df103934ec055d5 from peer 1d34e18a1d394d3f91c9f50d1c7bea4e ({username='slave'} at 127.29.219.68:40815)
I20240501 20:10:35.484196 4519 tablet_copy_service.cc:161] P 3e974c03d3c546f9ab6c91e762a5426b: Beginning new tablet copy session on tablet 786296114e8a47c68df103934ec055d5 from peer 1d34e18a1d394d3f91c9f50d1c7bea4e at {username='slave'} at 127.29.219.68:40815: session id = 1d34e18a1d394d3f91c9f50d1c7bea4e-786296114e8a47c68df103934ec055d5
I20240501 20:10:35.484812 12030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82260307259e4a9fb12231ff4abd79c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:35.486275 4519 tablet_copy_source_session.cc:205] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:35.487309 12038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 786296114e8a47c68df103934ec055d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:35.491665 12030 tablet_copy_client.cc:783] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:35.491963 12030 tablet_copy_client.cc:647] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:35.494217 12038 tablet_copy_client.cc:783] T 786296114e8a47c68df103934ec055d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:35.494432 12030 tablet_copy_client.cc:523] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:35.494458 12038 tablet_copy_client.cc:647] T 786296114e8a47c68df103934ec055d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:35.496284 12038 tablet_copy_client.cc:523] T 786296114e8a47c68df103934ec055d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:35.496594 4496 consensus_queue.cc:237] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:35.498021 12030 tablet_bootstrap.cc:492] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:35.499616 12038 tablet_bootstrap.cc:492] T 786296114e8a47c68df103934ec055d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:35.499626 4627 raft_consensus.cc:1261] T 4613582180754f729afc8d0b44356e97 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:35.499958 4433 consensus_peers.cc:485] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b -> Peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Couldn't send request to peer bceeae1c79964acabdae8ff771bf0c97. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4613582180754f729afc8d0b44356e97. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:35.500268 11050 consensus_queue.cc:1035] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:35.500684 4364 raft_consensus.cc:1261] T 4613582180754f729afc8d0b44356e97 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:35.501495 11050 consensus_queue.cc:1035] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:35.505507 12030 tablet_bootstrap.cc:492] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:35.506078 12030 tablet_bootstrap.cc:492] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd: Bootstrap complete.
I20240501 20:10:35.505800 11050 raft_consensus.cc:2932] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:35.506402 12030 ts_tablet_manager.cc:1367] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.001s
I20240501 20:10:35.507267 12030 raft_consensus.cc:348] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:35.507508 12030 raft_consensus.cc:729] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: LEARNER
I20240501 20:10:35.507769 12030 consensus_queue.cc:260] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:35.508410 12030 ts_tablet_manager.cc:1398] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:10:35.509603 4627 raft_consensus.cc:2932] T 4613582180754f729afc8d0b44356e97 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:35.510309 4254 tablet_copy_service.cc:342] P dca57823003e4729b769ca12367d7530: Request end of tablet copy session db0742ee3860468590eefef6ae6074fd-82260307259e4a9fb12231ff4abd79c5 received from {username='slave'} at 127.29.219.70:47215
I20240501 20:10:35.510422 4254 tablet_copy_service.cc:433] P dca57823003e4729b769ca12367d7530: ending tablet copy session db0742ee3860468590eefef6ae6074fd-82260307259e4a9fb12231ff4abd79c5 on tablet 82260307259e4a9fb12231ff4abd79c5 with peer db0742ee3860468590eefef6ae6074fd
I20240501 20:10:35.514504 4089 catalog_manager.cc:5484] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index -1 to 2, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:35.515071 12038 tablet_bootstrap.cc:492] T 786296114e8a47c68df103934ec055d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:35.515451 12038 tablet_bootstrap.cc:492] T 786296114e8a47c68df103934ec055d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap complete.
I20240501 20:10:35.515694 12038 ts_tablet_manager.cc:1367] T 786296114e8a47c68df103934ec055d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.004s
I20240501 20:10:35.516239 12052 ts_tablet_manager.cc:902] T ce0b97a1badb4935a4553902e6feb20b P db0742ee3860468590eefef6ae6074fd: Initiating tablet copy from peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:35.517143 12038 raft_consensus.cc:348] T 786296114e8a47c68df103934ec055d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:35.517315 12038 raft_consensus.cc:729] T 786296114e8a47c68df103934ec055d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: LEARNER
I20240501 20:10:35.517714 12038 consensus_queue.cc:260] T 786296114e8a47c68df103934ec055d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:35.518097 4364 raft_consensus.cc:2932] T 4613582180754f729afc8d0b44356e97 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } } }
I20240501 20:10:35.518821 12038 ts_tablet_manager.cc:1398] T 786296114e8a47c68df103934ec055d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 20:10:35.519613 4519 tablet_copy_service.cc:342] P 3e974c03d3c546f9ab6c91e762a5426b: Request end of tablet copy session 1d34e18a1d394d3f91c9f50d1c7bea4e-786296114e8a47c68df103934ec055d5 received from {username='slave'} at 127.29.219.68:40815
I20240501 20:10:35.519726 4519 tablet_copy_service.cc:433] P 3e974c03d3c546f9ab6c91e762a5426b: ending tablet copy session 1d34e18a1d394d3f91c9f50d1c7bea4e-786296114e8a47c68df103934ec055d5 on tablet 786296114e8a47c68df103934ec055d5 with peer 1d34e18a1d394d3f91c9f50d1c7bea4e
I20240501 20:10:35.527200 12053 ts_tablet_manager.cc:902] T 376ae6034f394a0fb0cab0d8a9241134 P 3e974c03d3c546f9ab6c91e762a5426b: Initiating tablet copy from peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:35.535593 12061 ts_tablet_manager.cc:902] T 92ce2c1ce3e0449bab5490b690721cbf P 10b47782558c41ee91f5e164a372049a: Initiating tablet copy from peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:35.537801 12052 tablet_copy_client.cc:308] T ce0b97a1badb4935a4553902e6feb20b P db0742ee3860468590eefef6ae6074fd: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.71:39447
I20240501 20:10:35.538771 7308 tablet_copy_service.cc:140] P bceeae1c79964acabdae8ff771bf0c97: Received BeginTabletCopySession request for tablet ce0b97a1badb4935a4553902e6feb20b from peer db0742ee3860468590eefef6ae6074fd ({username='slave'} at 127.29.219.70:36483)
I20240501 20:10:35.539003 7308 tablet_copy_service.cc:161] P bceeae1c79964acabdae8ff771bf0c97: Beginning new tablet copy session on tablet ce0b97a1badb4935a4553902e6feb20b from peer db0742ee3860468590eefef6ae6074fd at {username='slave'} at 127.29.219.70:36483: session id = db0742ee3860468590eefef6ae6074fd-ce0b97a1badb4935a4553902e6feb20b
I20240501 20:10:35.541718 7308 tablet_copy_source_session.cc:205] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:35.545727 12052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce0b97a1badb4935a4553902e6feb20b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:35.549961 12058 ts_tablet_manager.cc:902] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530: Initiating tablet copy from peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:35.556525 12052 tablet_copy_client.cc:783] T ce0b97a1badb4935a4553902e6feb20b P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:35.556856 12052 tablet_copy_client.cc:647] T ce0b97a1badb4935a4553902e6feb20b P db0742ee3860468590eefef6ae6074fd: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:35.558562 12052 tablet_copy_client.cc:523] T ce0b97a1badb4935a4553902e6feb20b P db0742ee3860468590eefef6ae6074fd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:35.568369 12052 tablet_bootstrap.cc:492] T ce0b97a1badb4935a4553902e6feb20b P db0742ee3860468590eefef6ae6074fd: Bootstrap starting.
I20240501 20:10:35.572227 12061 tablet_copy_client.cc:308] T 92ce2c1ce3e0449bab5490b690721cbf P 10b47782558c41ee91f5e164a372049a: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.70:36321
I20240501 20:10:35.573022 7175 tablet_copy_service.cc:140] P db0742ee3860468590eefef6ae6074fd: Received BeginTabletCopySession request for tablet 92ce2c1ce3e0449bab5490b690721cbf from peer 10b47782558c41ee91f5e164a372049a ({username='slave'} at 127.29.219.66:51075)
I20240501 20:10:35.573241 7175 tablet_copy_service.cc:161] P db0742ee3860468590eefef6ae6074fd: Beginning new tablet copy session on tablet 92ce2c1ce3e0449bab5490b690721cbf from peer 10b47782558c41ee91f5e164a372049a at {username='slave'} at 127.29.219.66:51075: session id = 10b47782558c41ee91f5e164a372049a-92ce2c1ce3e0449bab5490b690721cbf
I20240501 20:10:35.577168 7175 tablet_copy_source_session.cc:205] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:35.577569 12053 tablet_copy_client.cc:308] T 376ae6034f394a0fb0cab0d8a9241134 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.71:39447
I20240501 20:10:35.578230 7308 tablet_copy_service.cc:140] P bceeae1c79964acabdae8ff771bf0c97: Received BeginTabletCopySession request for tablet 376ae6034f394a0fb0cab0d8a9241134 from peer 3e974c03d3c546f9ab6c91e762a5426b ({username='slave'} at 127.29.219.67:40409)
I20240501 20:10:35.578398 7308 tablet_copy_service.cc:161] P bceeae1c79964acabdae8ff771bf0c97: Beginning new tablet copy session on tablet 376ae6034f394a0fb0cab0d8a9241134 from peer 3e974c03d3c546f9ab6c91e762a5426b at {username='slave'} at 127.29.219.67:40409: session id = 3e974c03d3c546f9ab6c91e762a5426b-376ae6034f394a0fb0cab0d8a9241134
I20240501 20:10:35.578704 12061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92ce2c1ce3e0449bab5490b690721cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:35.580686 7308 tablet_copy_source_session.cc:205] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:35.581708 12053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 376ae6034f394a0fb0cab0d8a9241134. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:35.587013 12053 tablet_copy_client.cc:783] T 376ae6034f394a0fb0cab0d8a9241134 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:35.587294 12053 tablet_copy_client.cc:647] T 376ae6034f394a0fb0cab0d8a9241134 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:35.589375 12053 tablet_copy_client.cc:523] T 376ae6034f394a0fb0cab0d8a9241134 P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:35.589721 12061 tablet_copy_client.cc:783] T 92ce2c1ce3e0449bab5490b690721cbf P 10b47782558c41ee91f5e164a372049a: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:35.589951 12061 tablet_copy_client.cc:647] T 92ce2c1ce3e0449bab5490b690721cbf P 10b47782558c41ee91f5e164a372049a: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:35.592250 12061 tablet_copy_client.cc:523] T 92ce2c1ce3e0449bab5490b690721cbf P 10b47782558c41ee91f5e164a372049a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:35.592322 12053 tablet_bootstrap.cc:492] T 376ae6034f394a0fb0cab0d8a9241134 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:35.596040 12061 tablet_bootstrap.cc:492] T 92ce2c1ce3e0449bab5490b690721cbf P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:35.623910 12061 tablet_bootstrap.cc:492] T 92ce2c1ce3e0449bab5490b690721cbf P 10b47782558c41ee91f5e164a372049a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:35.624573 12061 tablet_bootstrap.cc:492] T 92ce2c1ce3e0449bab5490b690721cbf P 10b47782558c41ee91f5e164a372049a: Bootstrap complete.
I20240501 20:10:35.629334 12058 tablet_copy_client.cc:308] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.69:46237
I20240501 20:10:35.630326 7042 tablet_copy_service.cc:140] P ef0a4ada548446b3a531cb39a3c4e38d: Received BeginTabletCopySession request for tablet cd21a0faa3bb487cb80f0951b12b3d67 from peer dca57823003e4729b769ca12367d7530 ({username='slave'} at 127.29.219.65:40207)
I20240501 20:10:35.630512 7042 tablet_copy_service.cc:161] P ef0a4ada548446b3a531cb39a3c4e38d: Beginning new tablet copy session on tablet cd21a0faa3bb487cb80f0951b12b3d67 from peer dca57823003e4729b769ca12367d7530 at {username='slave'} at 127.29.219.65:40207: session id = dca57823003e4729b769ca12367d7530-cd21a0faa3bb487cb80f0951b12b3d67
I20240501 20:10:35.633420 7042 tablet_copy_source_session.cc:205] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:35.634928 12058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd21a0faa3bb487cb80f0951b12b3d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:35.635291 12052 tablet_bootstrap.cc:492] T ce0b97a1badb4935a4553902e6feb20b P db0742ee3860468590eefef6ae6074fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:35.635954 12052 tablet_bootstrap.cc:492] T ce0b97a1badb4935a4553902e6feb20b P db0742ee3860468590eefef6ae6074fd: Bootstrap complete.
I20240501 20:10:35.636281 12052 ts_tablet_manager.cc:1367] T ce0b97a1badb4935a4553902e6feb20b P db0742ee3860468590eefef6ae6074fd: Time spent bootstrapping tablet: real 0.068s user 0.007s sys 0.002s
I20240501 20:10:35.637547 12061 ts_tablet_manager.cc:1367] T 92ce2c1ce3e0449bab5490b690721cbf P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.014s
I20240501 20:10:35.639802 12061 raft_consensus.cc:348] T 92ce2c1ce3e0449bab5490b690721cbf P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:35.640313 12061 raft_consensus.cc:729] T 92ce2c1ce3e0449bab5490b690721cbf P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: LEARNER
I20240501 20:10:35.637426 12052 raft_consensus.cc:348] T ce0b97a1badb4935a4553902e6feb20b P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:35.641193 12052 raft_consensus.cc:729] T ce0b97a1badb4935a4553902e6feb20b P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Initialized, Role: LEARNER
I20240501 20:10:35.641533 12052 consensus_queue.cc:260] T ce0b97a1badb4935a4553902e6feb20b P db0742ee3860468590eefef6ae6074fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: NON_VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: true } }
I20240501 20:10:35.640595 12061 consensus_queue.cc:260] T 92ce2c1ce3e0449bab5490b690721cbf P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:35.643579 12058 tablet_copy_client.cc:783] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:35.643829 12058 tablet_copy_client.cc:647] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:35.645385 12061 ts_tablet_manager.cc:1398] T 92ce2c1ce3e0449bab5490b690721cbf P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240501 20:10:35.646304 7175 tablet_copy_service.cc:342] P db0742ee3860468590eefef6ae6074fd: Request end of tablet copy session 10b47782558c41ee91f5e164a372049a-92ce2c1ce3e0449bab5490b690721cbf received from {username='slave'} at 127.29.219.66:51075
I20240501 20:10:35.646435 7175 tablet_copy_service.cc:433] P db0742ee3860468590eefef6ae6074fd: ending tablet copy session 10b47782558c41ee91f5e164a372049a-92ce2c1ce3e0449bab5490b690721cbf on tablet 92ce2c1ce3e0449bab5490b690721cbf with peer 10b47782558c41ee91f5e164a372049a
I20240501 20:10:35.648087 12058 tablet_copy_client.cc:523] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:35.651417 12058 tablet_bootstrap.cc:492] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:35.666047 12058 tablet_bootstrap.cc:492] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:35.666505 12058 tablet_bootstrap.cc:492] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530: Bootstrap complete.
I20240501 20:10:35.666811 12058 ts_tablet_manager.cc:1367] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20240501 20:10:35.667474 12058 raft_consensus.cc:348] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:35.667809 12058 raft_consensus.cc:729] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: LEARNER
I20240501 20:10:35.668037 12058 consensus_queue.cc:260] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:35.670140 12058 ts_tablet_manager.cc:1398] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 20:10:35.671507 7042 tablet_copy_service.cc:342] P ef0a4ada548446b3a531cb39a3c4e38d: Request end of tablet copy session dca57823003e4729b769ca12367d7530-cd21a0faa3bb487cb80f0951b12b3d67 received from {username='slave'} at 127.29.219.65:40207
I20240501 20:10:35.671643 7042 tablet_copy_service.cc:433] P ef0a4ada548446b3a531cb39a3c4e38d: ending tablet copy session dca57823003e4729b769ca12367d7530-cd21a0faa3bb487cb80f0951b12b3d67 on tablet cd21a0faa3bb487cb80f0951b12b3d67 with peer dca57823003e4729b769ca12367d7530
I20240501 20:10:35.690201 12052 ts_tablet_manager.cc:1398] T ce0b97a1badb4935a4553902e6feb20b P db0742ee3860468590eefef6ae6074fd: Time spent starting tablet: real 0.054s user 0.004s sys 0.000s
I20240501 20:10:35.700416 7308 tablet_copy_service.cc:342] P bceeae1c79964acabdae8ff771bf0c97: Request end of tablet copy session db0742ee3860468590eefef6ae6074fd-ce0b97a1badb4935a4553902e6feb20b received from {username='slave'} at 127.29.219.70:36483
I20240501 20:10:35.700593 7308 tablet_copy_service.cc:433] P bceeae1c79964acabdae8ff771bf0c97: ending tablet copy session db0742ee3860468590eefef6ae6074fd-ce0b97a1badb4935a4553902e6feb20b on tablet ce0b97a1badb4935a4553902e6feb20b with peer db0742ee3860468590eefef6ae6074fd
I20240501 20:10:35.702931 12053 tablet_bootstrap.cc:492] T 376ae6034f394a0fb0cab0d8a9241134 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:35.703327 12053 tablet_bootstrap.cc:492] T 376ae6034f394a0fb0cab0d8a9241134 P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap complete.
I20240501 20:10:35.703394 12061 ts_tablet_manager.cc:902] T 9709a8bb107742b79078f91da9fab8f6 P 10b47782558c41ee91f5e164a372049a: Initiating tablet copy from peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:35.703583 12053 ts_tablet_manager.cc:1367] T 376ae6034f394a0fb0cab0d8a9241134 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.111s user 0.053s sys 0.046s
I20240501 20:10:35.704317 12053 raft_consensus.cc:348] T 376ae6034f394a0fb0cab0d8a9241134 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:35.704540 12053 raft_consensus.cc:729] T 376ae6034f394a0fb0cab0d8a9241134 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: LEARNER
I20240501 20:10:35.704550 7152 consensus_queue.cc:237] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:35.704702 12053 consensus_queue.cc:260] T 376ae6034f394a0fb0cab0d8a9241134 P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:35.707147 12053 ts_tablet_manager.cc:1398] T 376ae6034f394a0fb0cab0d8a9241134 P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:35.709669 7308 tablet_copy_service.cc:342] P bceeae1c79964acabdae8ff771bf0c97: Request end of tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-376ae6034f394a0fb0cab0d8a9241134 received from {username='slave'} at 127.29.219.67:40409
I20240501 20:10:35.709870 7308 tablet_copy_service.cc:433] P bceeae1c79964acabdae8ff771bf0c97: ending tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-376ae6034f394a0fb0cab0d8a9241134 on tablet 376ae6034f394a0fb0cab0d8a9241134 with peer 3e974c03d3c546f9ab6c91e762a5426b
W20240501 20:10:35.720914 7090 consensus_peers.cc:485] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67703d84deb9417bb21bde6365deda2f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:35.721964 4631 raft_consensus.cc:1261] T 67703d84deb9417bb21bde6365deda2f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:35.724822 11016 consensus_queue.cc:1035] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:35.733428 7419 raft_consensus.cc:1261] T 67703d84deb9417bb21bde6365deda2f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:35.734328 11016 consensus_queue.cc:1035] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:35.734910 12058 ts_tablet_manager.cc:902] T 9c77905da1b3453fa9a83366f7c51c37 P dca57823003e4729b769ca12367d7530: Initiating tablet copy from peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:35.752780 11920 raft_consensus.cc:2932] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:35.753545 7419 raft_consensus.cc:2932] T 67703d84deb9417bb21bde6365deda2f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:35.759521 4627 raft_consensus.cc:2932] T 67703d84deb9417bb21bde6365deda2f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:35.766942 7022 raft_consensus.cc:1203] T 024c925714cb419aaa8870a053a6ca2f P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:35.772174 4089 catalog_manager.cc:5484] T 67703d84deb9417bb21bde6365deda2f P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: config changed from index -1 to 2, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:35.811206 12061 tablet_copy_client.cc:308] T 9709a8bb107742b79078f91da9fab8f6 P 10b47782558c41ee91f5e164a372049a: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.70:36321
I20240501 20:10:35.813262 7175 tablet_copy_service.cc:140] P db0742ee3860468590eefef6ae6074fd: Received BeginTabletCopySession request for tablet 9709a8bb107742b79078f91da9fab8f6 from peer 10b47782558c41ee91f5e164a372049a ({username='slave'} at 127.29.219.66:51075)
I20240501 20:10:35.813421 7175 tablet_copy_service.cc:161] P db0742ee3860468590eefef6ae6074fd: Beginning new tablet copy session on tablet 9709a8bb107742b79078f91da9fab8f6 from peer 10b47782558c41ee91f5e164a372049a at {username='slave'} at 127.29.219.66:51075: session id = 10b47782558c41ee91f5e164a372049a-9709a8bb107742b79078f91da9fab8f6
I20240501 20:10:35.816322 7175 tablet_copy_source_session.cc:205] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:35.817586 12061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9709a8bb107742b79078f91da9fab8f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:35.824498 12058 tablet_copy_client.cc:308] T 9c77905da1b3453fa9a83366f7c51c37 P dca57823003e4729b769ca12367d7530: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.72:39817
I20240501 20:10:35.825369 12038 ts_tablet_manager.cc:902] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Initiating tablet copy from peer b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:35.825830 7441 tablet_copy_service.cc:140] P b9957d4086bc4a76b18ed9265a5739d5: Received BeginTabletCopySession request for tablet 9c77905da1b3453fa9a83366f7c51c37 from peer dca57823003e4729b769ca12367d7530 ({username='slave'} at 127.29.219.65:37513)
I20240501 20:10:35.826009 7441 tablet_copy_service.cc:161] P b9957d4086bc4a76b18ed9265a5739d5: Beginning new tablet copy session on tablet 9c77905da1b3453fa9a83366f7c51c37 from peer dca57823003e4729b769ca12367d7530 at {username='slave'} at 127.29.219.65:37513: session id = dca57823003e4729b769ca12367d7530-9c77905da1b3453fa9a83366f7c51c37
I20240501 20:10:35.827759 7419 raft_consensus.cc:1203] T 1d3bb8370e7c41dc9cf7f1c835ac275b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:35.828881 7441 tablet_copy_source_session.cc:205] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:35.833276 12061 tablet_copy_client.cc:783] T 9709a8bb107742b79078f91da9fab8f6 P 10b47782558c41ee91f5e164a372049a: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:35.833588 12038 tablet_copy_client.cc:308] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.72:39817
I20240501 20:10:35.833636 12061 tablet_copy_client.cc:647] T 9709a8bb107742b79078f91da9fab8f6 P 10b47782558c41ee91f5e164a372049a: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:35.834240 7441 tablet_copy_service.cc:140] P b9957d4086bc4a76b18ed9265a5739d5: Received BeginTabletCopySession request for tablet 9e0ecc1043af4bc1ab31dfa80ea47bb4 from peer 1d34e18a1d394d3f91c9f50d1c7bea4e ({username='slave'} at 127.29.219.68:34551)
I20240501 20:10:35.834416 7441 tablet_copy_service.cc:161] P b9957d4086bc4a76b18ed9265a5739d5: Beginning new tablet copy session on tablet 9e0ecc1043af4bc1ab31dfa80ea47bb4 from peer 1d34e18a1d394d3f91c9f50d1c7bea4e at {username='slave'} at 127.29.219.68:34551: session id = 1d34e18a1d394d3f91c9f50d1c7bea4e-9e0ecc1043af4bc1ab31dfa80ea47bb4
I20240501 20:10:35.836972 7441 tablet_copy_source_session.cc:205] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:35.838395 12038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e0ecc1043af4bc1ab31dfa80ea47bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:35.841437 12058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c77905da1b3453fa9a83366f7c51c37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:35.843116 12061 tablet_copy_client.cc:523] T 9709a8bb107742b79078f91da9fab8f6 P 10b47782558c41ee91f5e164a372049a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:35.848244 12038 tablet_copy_client.cc:783] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:35.848546 12038 tablet_copy_client.cc:647] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:35.852140 12058 tablet_copy_client.cc:783] T 9c77905da1b3453fa9a83366f7c51c37 P dca57823003e4729b769ca12367d7530: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:35.852413 12058 tablet_copy_client.cc:647] T 9c77905da1b3453fa9a83366f7c51c37 P dca57823003e4729b769ca12367d7530: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:35.853367 12038 tablet_copy_client.cc:523] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:35.856501 12058 tablet_copy_client.cc:523] T 9c77905da1b3453fa9a83366f7c51c37 P dca57823003e4729b769ca12367d7530: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:35.858587 12038 tablet_bootstrap.cc:492] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:35.861407 12061 tablet_bootstrap.cc:492] T 9709a8bb107742b79078f91da9fab8f6 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:35.863282 4627 consensus_queue.cc:237] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:35.863955 12058 tablet_bootstrap.cc:492] T 9c77905da1b3453fa9a83366f7c51c37 P dca57823003e4729b769ca12367d7530: Bootstrap starting.
I20240501 20:10:35.868041 12017 ts_tablet_manager.cc:902] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5: Initiating tablet copy from peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:35.872532 12038 tablet_bootstrap.cc:492] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:35.872982 12038 tablet_bootstrap.cc:492] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap complete.
I20240501 20:10:35.873265 12038 ts_tablet_manager.cc:1367] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.004s
I20240501 20:10:35.874425 12038 raft_consensus.cc:348] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:35.874737 12038 raft_consensus.cc:729] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: LEARNER
I20240501 20:10:35.875025 12105 ts_tablet_manager.cc:902] T 56b2082919b548bfb56b939373caf770 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Initiating tablet copy from peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
W20240501 20:10:35.875294 4564 consensus_peers.cc:485] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ffe6b742c3743d8b7de7a0c4105637f. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:35.875003 12038 consensus_queue.cc:260] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:35.875952 12038 ts_tablet_manager.cc:1398] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20240501 20:10:35.876859 7153 raft_consensus.cc:1261] T 3ffe6b742c3743d8b7de7a0c4105637f P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:35.877266 4232 raft_consensus.cc:1261] T 3ffe6b742c3743d8b7de7a0c4105637f P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:35.877326 11901 consensus_queue.cc:1035] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:35.877875 7441 tablet_copy_service.cc:342] P b9957d4086bc4a76b18ed9265a5739d5: Request end of tablet copy session 1d34e18a1d394d3f91c9f50d1c7bea4e-9e0ecc1043af4bc1ab31dfa80ea47bb4 received from {username='slave'} at 127.29.219.68:34551
I20240501 20:10:35.878012 7441 tablet_copy_service.cc:433] P b9957d4086bc4a76b18ed9265a5739d5: ending tablet copy session 1d34e18a1d394d3f91c9f50d1c7bea4e-9e0ecc1043af4bc1ab31dfa80ea47bb4 on tablet 9e0ecc1043af4bc1ab31dfa80ea47bb4 with peer 1d34e18a1d394d3f91c9f50d1c7bea4e
I20240501 20:10:35.878047 11901 consensus_queue.cc:1035] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:35.882061 11901 raft_consensus.cc:2932] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:35.887120 12058 tablet_bootstrap.cc:492] T 9c77905da1b3453fa9a83366f7c51c37 P dca57823003e4729b769ca12367d7530: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:35.887567 12058 tablet_bootstrap.cc:492] T 9c77905da1b3453fa9a83366f7c51c37 P dca57823003e4729b769ca12367d7530: Bootstrap complete.
I20240501 20:10:35.887861 12058 ts_tablet_manager.cc:1367] T 9c77905da1b3453fa9a83366f7c51c37 P dca57823003e4729b769ca12367d7530: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.005s
I20240501 20:10:35.888672 12058 raft_consensus.cc:348] T 9c77905da1b3453fa9a83366f7c51c37 P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:35.891175 7153 raft_consensus.cc:2932] T 3ffe6b742c3743d8b7de7a0c4105637f P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:35.889655 4232 raft_consensus.cc:2932] T 3ffe6b742c3743d8b7de7a0c4105637f P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } }
I20240501 20:10:35.891939 12058 raft_consensus.cc:729] T 9c77905da1b3453fa9a83366f7c51c37 P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dca57823003e4729b769ca12367d7530, State: Initialized, Role: LEARNER
I20240501 20:10:35.892187 12058 consensus_queue.cc:260] T 9c77905da1b3453fa9a83366f7c51c37 P dca57823003e4729b769ca12367d7530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: NON_VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: true } }
I20240501 20:10:35.893159 12058 ts_tablet_manager.cc:1398] T 9c77905da1b3453fa9a83366f7c51c37 P dca57823003e4729b769ca12367d7530: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240501 20:10:35.893849 4091 catalog_manager.cc:5484] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index -1 to 2, NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) added. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:35.902076 7441 tablet_copy_service.cc:342] P b9957d4086bc4a76b18ed9265a5739d5: Request end of tablet copy session dca57823003e4729b769ca12367d7530-9c77905da1b3453fa9a83366f7c51c37 received from {username='slave'} at 127.29.219.65:37513
I20240501 20:10:35.902235 7441 tablet_copy_service.cc:433] P b9957d4086bc4a76b18ed9265a5739d5: ending tablet copy session dca57823003e4729b769ca12367d7530-9c77905da1b3453fa9a83366f7c51c37 on tablet 9c77905da1b3453fa9a83366f7c51c37 with peer dca57823003e4729b769ca12367d7530
I20240501 20:10:35.922719 12017 tablet_copy_client.cc:308] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.69:46237
I20240501 20:10:35.923605 7042 tablet_copy_service.cc:140] P ef0a4ada548446b3a531cb39a3c4e38d: Received BeginTabletCopySession request for tablet 99bd0e9f4cd7496785ad71480d7d3d61 from peer b9957d4086bc4a76b18ed9265a5739d5 ({username='slave'} at 127.29.219.72:37221)
I20240501 20:10:35.923679 12061 tablet_bootstrap.cc:492] T 9709a8bb107742b79078f91da9fab8f6 P 10b47782558c41ee91f5e164a372049a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:35.923760 7042 tablet_copy_service.cc:161] P ef0a4ada548446b3a531cb39a3c4e38d: Beginning new tablet copy session on tablet 99bd0e9f4cd7496785ad71480d7d3d61 from peer b9957d4086bc4a76b18ed9265a5739d5 at {username='slave'} at 127.29.219.72:37221: session id = b9957d4086bc4a76b18ed9265a5739d5-99bd0e9f4cd7496785ad71480d7d3d61
I20240501 20:10:35.924024 12061 tablet_bootstrap.cc:492] T 9709a8bb107742b79078f91da9fab8f6 P 10b47782558c41ee91f5e164a372049a: Bootstrap complete.
I20240501 20:10:35.926213 12061 ts_tablet_manager.cc:1367] T 9709a8bb107742b79078f91da9fab8f6 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.065s user 0.008s sys 0.000s
I20240501 20:10:35.926493 7042 tablet_copy_source_session.cc:205] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:35.927049 12061 raft_consensus.cc:348] T 9709a8bb107742b79078f91da9fab8f6 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:35.927284 12061 raft_consensus.cc:729] T 9709a8bb107742b79078f91da9fab8f6 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: LEARNER
I20240501 20:10:35.927459 12061 consensus_queue.cc:260] T 9709a8bb107742b79078f91da9fab8f6 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:35.928102 12061 ts_tablet_manager.cc:1398] T 9709a8bb107742b79078f91da9fab8f6 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:35.928781 7175 tablet_copy_service.cc:342] P db0742ee3860468590eefef6ae6074fd: Request end of tablet copy session 10b47782558c41ee91f5e164a372049a-9709a8bb107742b79078f91da9fab8f6 received from {username='slave'} at 127.29.219.66:51075
I20240501 20:10:35.929013 7175 tablet_copy_service.cc:433] P db0742ee3860468590eefef6ae6074fd: ending tablet copy session 10b47782558c41ee91f5e164a372049a-9709a8bb107742b79078f91da9fab8f6 on tablet 9709a8bb107742b79078f91da9fab8f6 with peer 10b47782558c41ee91f5e164a372049a
I20240501 20:10:35.931645 12017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99bd0e9f4cd7496785ad71480d7d3d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:35.939306 12017 tablet_copy_client.cc:783] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:35.939579 12017 tablet_copy_client.cc:647] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:35.941826 12017 tablet_copy_client.cc:523] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:35.945984 12017 tablet_bootstrap.cc:492] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap starting.
I20240501 20:10:35.968891 4365 raft_consensus.cc:1203] T 92ce2c1ce3e0449bab5490b690721cbf P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:35.982499 12017 tablet_bootstrap.cc:492] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:35.982969 12017 tablet_bootstrap.cc:492] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5: Bootstrap complete.
I20240501 20:10:35.983273 12017 ts_tablet_manager.cc:1367] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5: Time spent bootstrapping tablet: real 0.037s user 0.011s sys 0.001s
I20240501 20:10:35.984066 12017 raft_consensus.cc:348] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:35.984313 12017 raft_consensus.cc:729] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Initialized, Role: LEARNER
I20240501 20:10:35.984509 12017 consensus_queue.cc:260] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:35.985209 12017 ts_tablet_manager.cc:1398] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:35.986413 7042 tablet_copy_service.cc:342] P ef0a4ada548446b3a531cb39a3c4e38d: Request end of tablet copy session b9957d4086bc4a76b18ed9265a5739d5-99bd0e9f4cd7496785ad71480d7d3d61 received from {username='slave'} at 127.29.219.72:37221
I20240501 20:10:35.986546 7042 tablet_copy_service.cc:433] P ef0a4ada548446b3a531cb39a3c4e38d: ending tablet copy session b9957d4086bc4a76b18ed9265a5739d5-99bd0e9f4cd7496785ad71480d7d3d61 on tablet 99bd0e9f4cd7496785ad71480d7d3d61 with peer b9957d4086bc4a76b18ed9265a5739d5
I20240501 20:10:35.997174 12125 ts_tablet_manager.cc:902] T 4613582180754f729afc8d0b44356e97 P bceeae1c79964acabdae8ff771bf0c97: Initiating tablet copy from peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:36.002010 4627 raft_consensus.cc:1203] T 786296114e8a47c68df103934ec055d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:36.001943 7022 consensus_queue.cc:237] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:36.011284 7155 raft_consensus.cc:1203] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 20:10:36.033367 6958 consensus_peers.cc:485] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d -> Peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Couldn't send request to peer 1d34e18a1d394d3f91c9f50d1c7bea4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79af181b5f014a40a4c123b47a720dec. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:36.035400 4499 raft_consensus.cc:1261] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:36.036145 4365 raft_consensus.cc:1261] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 20:10:36.036276 11840 consensus_queue.cc:1035] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:36.038779 11840 consensus_queue.cc:1035] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:36.051139 11840 raft_consensus.cc:2932] T 79af181b5f014a40a4c123b47a720dec P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:36.061614 4232 raft_consensus.cc:1203] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:36.063084 4496 raft_consensus.cc:1203] T 376ae6034f394a0fb0cab0d8a9241134 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:36.077406 4364 raft_consensus.cc:2932] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:36.079346 4499 raft_consensus.cc:2932] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:36.095402 4091 catalog_manager.cc:5484] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index 4 to 5, NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) added. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } } }
I20240501 20:10:36.120554 7153 raft_consensus.cc:1203] T ce0b97a1badb4935a4553902e6feb20b P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:36.147406 12105 tablet_copy_client.cc:308] T 56b2082919b548bfb56b939373caf770 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.69:46237
I20240501 20:10:36.148069 7042 tablet_copy_service.cc:140] P ef0a4ada548446b3a531cb39a3c4e38d: Received BeginTabletCopySession request for tablet 56b2082919b548bfb56b939373caf770 from peer 1d34e18a1d394d3f91c9f50d1c7bea4e ({username='slave'} at 127.29.219.68:40675)
I20240501 20:10:36.148276 7042 tablet_copy_service.cc:161] P ef0a4ada548446b3a531cb39a3c4e38d: Beginning new tablet copy session on tablet 56b2082919b548bfb56b939373caf770 from peer 1d34e18a1d394d3f91c9f50d1c7bea4e at {username='slave'} at 127.29.219.68:40675: session id = 1d34e18a1d394d3f91c9f50d1c7bea4e-56b2082919b548bfb56b939373caf770
I20240501 20:10:36.152313 7042 tablet_copy_source_session.cc:205] T 56b2082919b548bfb56b939373caf770 P ef0a4ada548446b3a531cb39a3c4e38d: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:36.153802 12105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56b2082919b548bfb56b939373caf770. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:36.160468 12105 tablet_copy_client.cc:783] T 56b2082919b548bfb56b939373caf770 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:36.160748 12105 tablet_copy_client.cc:647] T 56b2082919b548bfb56b939373caf770 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:36.162122 12125 tablet_copy_client.cc:308] T 4613582180754f729afc8d0b44356e97 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.67:34595
I20240501 20:10:36.162844 4519 tablet_copy_service.cc:140] P 3e974c03d3c546f9ab6c91e762a5426b: Received BeginTabletCopySession request for tablet 4613582180754f729afc8d0b44356e97 from peer bceeae1c79964acabdae8ff771bf0c97 ({username='slave'} at 127.29.219.71:60861)
I20240501 20:10:36.162981 4519 tablet_copy_service.cc:161] P 3e974c03d3c546f9ab6c91e762a5426b: Beginning new tablet copy session on tablet 4613582180754f729afc8d0b44356e97 from peer bceeae1c79964acabdae8ff771bf0c97 at {username='slave'} at 127.29.219.71:60861: session id = bceeae1c79964acabdae8ff771bf0c97-4613582180754f729afc8d0b44356e97
I20240501 20:10:36.164880 12105 tablet_copy_client.cc:523] T 56b2082919b548bfb56b939373caf770 P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:36.165726 4519 tablet_copy_source_session.cc:205] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:36.167097 12125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4613582180754f729afc8d0b44356e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:36.171001 12105 tablet_bootstrap.cc:492] T 56b2082919b548bfb56b939373caf770 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:36.173612 12125 tablet_copy_client.cc:783] T 4613582180754f729afc8d0b44356e97 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:36.173949 12125 tablet_copy_client.cc:647] T 4613582180754f729afc8d0b44356e97 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:36.175617 12125 tablet_copy_client.cc:523] T 4613582180754f729afc8d0b44356e97 P bceeae1c79964acabdae8ff771bf0c97: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:36.178486 12125 tablet_bootstrap.cc:492] T 4613582180754f729afc8d0b44356e97 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap starting.
I20240501 20:10:36.184418 12105 tablet_bootstrap.cc:492] T 56b2082919b548bfb56b939373caf770 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:36.184739 12105 tablet_bootstrap.cc:492] T 56b2082919b548bfb56b939373caf770 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap complete.
I20240501 20:10:36.184919 12105 ts_tablet_manager.cc:1367] T 56b2082919b548bfb56b939373caf770 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.003s
I20240501 20:10:36.185583 12105 raft_consensus.cc:348] T 56b2082919b548bfb56b939373caf770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:36.185806 12105 raft_consensus.cc:729] T 56b2082919b548bfb56b939373caf770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: LEARNER
I20240501 20:10:36.185968 12105 consensus_queue.cc:260] T 56b2082919b548bfb56b939373caf770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:36.186511 12105 ts_tablet_manager.cc:1398] T 56b2082919b548bfb56b939373caf770 P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:36.188290 7042 tablet_copy_service.cc:342] P ef0a4ada548446b3a531cb39a3c4e38d: Request end of tablet copy session 1d34e18a1d394d3f91c9f50d1c7bea4e-56b2082919b548bfb56b939373caf770 received from {username='slave'} at 127.29.219.68:40675
I20240501 20:10:36.188434 7042 tablet_copy_service.cc:433] P ef0a4ada548446b3a531cb39a3c4e38d: ending tablet copy session 1d34e18a1d394d3f91c9f50d1c7bea4e-56b2082919b548bfb56b939373caf770 on tablet 56b2082919b548bfb56b939373caf770 with peer 1d34e18a1d394d3f91c9f50d1c7bea4e
I20240501 20:10:36.192528 12125 tablet_bootstrap.cc:492] T 4613582180754f729afc8d0b44356e97 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:36.193207 12125 tablet_bootstrap.cc:492] T 4613582180754f729afc8d0b44356e97 P bceeae1c79964acabdae8ff771bf0c97: Bootstrap complete.
I20240501 20:10:36.193511 12125 ts_tablet_manager.cc:1367] T 4613582180754f729afc8d0b44356e97 P bceeae1c79964acabdae8ff771bf0c97: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20240501 20:10:36.194293 12125 raft_consensus.cc:348] T 4613582180754f729afc8d0b44356e97 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:36.194512 12125 raft_consensus.cc:729] T 4613582180754f729afc8d0b44356e97 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Initialized, Role: LEARNER
I20240501 20:10:36.194844 12125 consensus_queue.cc:260] T 4613582180754f729afc8d0b44356e97 P bceeae1c79964acabdae8ff771bf0c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: NON_VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: true } }
I20240501 20:10:36.195384 12125 ts_tablet_manager.cc:1398] T 4613582180754f729afc8d0b44356e97 P bceeae1c79964acabdae8ff771bf0c97: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:36.198889 4519 tablet_copy_service.cc:342] P 3e974c03d3c546f9ab6c91e762a5426b: Request end of tablet copy session bceeae1c79964acabdae8ff771bf0c97-4613582180754f729afc8d0b44356e97 received from {username='slave'} at 127.29.219.71:60861
I20240501 20:10:36.198994 4519 tablet_copy_service.cc:433] P 3e974c03d3c546f9ab6c91e762a5426b: ending tablet copy session bceeae1c79964acabdae8ff771bf0c97-4613582180754f729afc8d0b44356e97 on tablet 4613582180754f729afc8d0b44356e97 with peer bceeae1c79964acabdae8ff771bf0c97
I20240501 20:10:36.221695 11816 raft_consensus.cc:1050] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd: attempting to promote NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d to VOTER
I20240501 20:10:36.222563 11816 consensus_queue.cc:237] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:36.239645 7019 raft_consensus.cc:1261] T 024c925714cb419aaa8870a053a6ca2f P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEARNER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.240309 7281 raft_consensus.cc:1261] T 024c925714cb419aaa8870a053a6ca2f P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.241860 11016 consensus_queue.cc:1035] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.244217 11920 consensus_queue.cc:1035] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.258046 4232 raft_consensus.cc:1261] T 024c925714cb419aaa8870a053a6ca2f P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.258846 11816 consensus_queue.cc:1035] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.264243 11816 raft_consensus.cc:2932] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:36.270360 4364 raft_consensus.cc:1203] T 9709a8bb107742b79078f91da9fab8f6 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:36.273010 7019 raft_consensus.cc:2932] T 024c925714cb419aaa8870a053a6ca2f P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:36.278121 7281 raft_consensus.cc:2932] T 024c925714cb419aaa8870a053a6ca2f P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:36.282929 4090 catalog_manager.cc:5484] T 024c925714cb419aaa8870a053a6ca2f P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: config changed from index 2 to 3, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:36.297858 12144 ts_tablet_manager.cc:902] T 67703d84deb9417bb21bde6365deda2f P ef0a4ada548446b3a531cb39a3c4e38d: Initiating tablet copy from peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
F20240501 20:10:36.298414 12144 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-024c925714cb419aaa8870a053a6ca2f.queue_time_us, tablet-download-pool-024c925714cb419aaa8870a053a6ca2f.queue_time_us) (tablet-download-pool-024c925714cb419aaa8870a053a6ca2f.run_cpu_time_us, tablet-download-pool-024c925714cb419aaa8870a053a6ca2f.run_cpu_time_us) (tablet-download-pool-024c925714cb419aaa8870a053a6ca2f.run_wall_time_us, tablet-download-pool-024c925714cb419aaa8870a053a6ca2f.run_wall_time_us) (tablet-download-pool-04a5fb2b808d466491144e64eb54d0fd.queue_time_us, tablet-download-pool-04a5fb2b808d466491144e64eb54d0fd.queue_time_us) (tablet-download-pool-04a5fb2b808d466491144e64eb54d0fd.run_cpu_time_us, tablet-download-pool-04a5fb2b808d466491144e64eb54d0fd.run_cpu_time_us) (tablet-download-pool-04a5fb2b808d466491144e64eb54d0fd.run_wall_time_us, tablet-download-pool-04a5fb2b808d466491144e64eb54d0fd.run_wall_time_us) (tablet-download-pool-1c6c89fa7e6f485e889a5197da4bd135.queue_time_us, tablet-download-pool-1c6c89fa7e6f485e889a5197da4bd135.queue_time_us) (tablet-download-pool-1c6c89fa7e6f485e889a5197da4bd135.run_cpu_time_us, tablet-download-pool-1c6c89fa7e6f485e889a5197da4bd135.run_cpu_time_us) (tablet-download-pool-1c6c89fa7e6f485e889a5197da4bd135.run_wall_time_us, tablet-download-pool-1c6c89fa7e6f485e889a5197da4bd135.run_wall_time_us) (tablet-download-pool-25f01a01106f47c489e252c928cf28bb.queue_time_us, tablet-download-pool-25f01a01106f47c489e252c928cf28bb.queue_time_us) (tablet-download-pool-25f01a01106f47c489e252c928cf28bb.run_cpu_time_us, tablet-download-pool-25f01a01106f47c489e252c928cf28bb.run_cpu_time_us) (tablet-download-pool-25f01a01106f47c489e252c928cf28bb.run_wall_time_us, tablet-download-pool-25f01a01106f47c489e252c928cf28bb.run_wall_time_us) (tablet-download-pool-2aa1a5f266e840698f2127289628dcba.queue_time_us, tablet-download-pool-2aa1a5f266e840698f2127289628dcba.queue_time_us) (tablet-download-pool-2aa1a5f266e840698f2127289628dcba.run_cpu_time_us, tablet-download-pool-2aa1a5f266e840698f2127289628dcba.run_cpu_time_us) (tablet-download-pool-2aa1a5f266e840698f2127289628dcba.run_wall_time_us, tablet-download-pool-2aa1a5f266e840698f2127289628dcba.run_wall_time_us) (tablet-download-pool-46df0faa681b469abe9e383b181a6053.queue_time_us, tablet-download-pool-46df0faa681b469abe9e383b181a6053.queue_time_us) (tablet-download-pool-46df0faa681b469abe9e383b181a6053.run_cpu_time_us, tablet-download-pool-46df0faa681b469abe9e383b181a6053.run_cpu_time_us) (tablet-download-pool-46df0faa681b469abe9e383b181a6053.run_wall_time_us, tablet-download-pool-46df0faa681b469abe9e383b181a6053.run_wall_time_us) (tablet-download-pool-644d4d293e8c462297c6419965bd2c89.queue_time_us, tablet-download-pool-644d4d293e8c462297c6419965bd2c89.queue_time_us) (tablet-download-pool-644d4d293e8c462297c6419965bd2c89.run_cpu_time_us, tablet-download-pool-644d4d293e8c462297c6419965bd2c89.run_cpu_time_us) (tablet-download-pool-644d4d293e8c462297c6419965bd2c89.run_wall_time_us, tablet-download-pool-644d4d293e8c462297c6419965bd2c89.run_wall_time_us) (tablet-download-pool-67703d84deb9417bb21bde6365deda2f.queue_time_us, tablet-download-pool-67703d84deb9417bb21bde6365deda2f.queue_time_us) (tablet-download-pool-797b2374d4214c8eb39982c339a7fbad.queue_time_us, tablet-download-pool-797b2374d4214c8eb39982c339a7fbad.queue_time_us) (tablet-download-pool-797b2374d4214c8eb39982c339a7fbad.run_cpu_time_us, tablet-download-pool-797b2374d4214c8eb39982c339a7fbad.run_cpu_time_us) (tablet-download-pool-797b2374d4214c8eb39982c339a7fbad.run_wall_time_us, tablet-download-pool-797b2374d4214c8eb39982c339a7fbad.run_wall_time_us) (tablet-download-pool-7bcb854a8f7545cc86323abd69c26c23.queue_time_us, tablet-download-pool-7bcb854a8f7545cc86323abd69c26c23.queue_time_us) (tablet-download-pool-7bcb854a8f7545cc86323abd69c26c23.run_cpu_time_us, tablet-download-pool-7bcb854a8f7545cc86323abd69c26c23.run_cpu_time_us) (tablet-download-pool-7bcb854a8f7545cc86323abd69c26c23.run_wall_time_us, tablet-download-pool-7bcb854a8f7545cc86323abd69c26c23.run_wall_time_us) (tablet-download-pool-9d634c1feb7449f686cf3af2a07181e0.queue_time_us, tablet-download-pool-9d634c1feb7449f686cf3af2a07181e0.queue_time_us) (tablet-download-pool-9d634c1feb7449f686cf3af2a07181e0.run_cpu_time_us, tablet-download-pool-9d634c1feb7449f686cf3af2a07181e0.run_cpu_time_us) (tablet-download-pool-9d634c1feb7449f686cf3af2a07181e0.run_wall_time_us, tablet-download-pool-9d634c1feb7449f686cf3af2a07181e0.run_wall_time_us) (tablet-download-pool-a20287c2cab44a62b28037a463d4b49b.queue_time_us, tablet-download-pool-a20287c2cab44a62b28037a463d4b49b.queue_time_us) (tablet-download-pool-a20287c2cab44a62b28037a463d4b49b.run_cpu_time_us, tablet-download-pool-a20287c2cab44a62b28037a463d4b49b.run_cpu_time_us) (tablet-download-pool-a20287c2cab44a62b28037a463d4b49b.run_wall_time_us, tablet-download-pool-a20287c2cab44a62b28037a463d4b49b.run_wall_time_us) (tablet-download-pool-dfaa3794205d482c90aa4f860fa99cf3.queue_time_us, tablet-download-pool-dfaa3794205d482c90aa4f860fa99cf3.queue_time_us) (tablet-download-pool-dfaa3794205d482c90aa4f860fa99cf3.run_cpu_time_us, tablet-download-pool-dfaa3794205d482c90aa4f860fa99cf3.run_cpu_time_us) (tablet-download-pool-dfaa3794205d482c90aa4f860fa99cf3.run_wall_time_us, tablet-download-pool-dfaa3794205d482c90aa4f860fa99cf3.run_wall_time_us) (tablet-download-pool-f18af954cefa48d4882313fe18abbd27.queue_time_us, tablet-download-pool-f18af954cefa48d4882313fe18abbd27.queue_time_us) (tablet-download-pool-f18af954cefa48d4882313fe18abbd27.run_cpu_time_us, tablet-download-pool-f18af954cefa48d4882313fe18abbd27.run_cpu_time_us) (tablet-download-pool-f18af954cefa48d4882313fe18abbd27.run_wall_time_us, tablet-download-pool-f18af954cefa48d4882313fe18abbd27.run_wall_time_us) (tablet-download-pool-fcff4be357084bf0903cfb1c0f62a866.queue_time_us, tablet-download-pool-fcff4be357084bf0903cfb1c0f62a866.queue_time_us) (tablet-download-pool-fcff4be357084bf0903cfb1c0f62a866.run_cpu_time_us, tablet-download-pool-fcff4be357084bf0903cfb1c0f62a866.run_cpu_time_us) (tablet-download-pool-fcff4be357084bf0903cfb1c0f62a866.run_wall_time_us, tablet-download-pool-fcff4be357084bf0903cfb1c0f62a866.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f8c27e94dcd google::LogMessage::Fail() at ??:0
@ 0x7f8c27e98b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f8c27e947cc google::LogMessage::Flush() at ??:0
@ 0x7f8c27e95f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8c2abd5742 kudu::TraceMetrics::InternName() at ??:0
I20240501 20:10:36.351939 4234 raft_consensus.cc:1203] T 9c77905da1b3453fa9a83366f7c51c37 P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:36.353771 4627 raft_consensus.cc:1203] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:36.360422 7417 raft_consensus.cc:1203] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
@ 0x7f8c2aba1f8b kudu::ThreadPool::ThreadPool() at ??:0
I20240501 20:10:36.369829 7281 raft_consensus.cc:1203] T 4613582180754f729afc8d0b44356e97 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:36.379204 4232 raft_consensus.cc:2932] T 024c925714cb419aaa8870a053a6ca2f P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:36.382244 7153 consensus_queue.cc:237] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
@ 0x7f8c2ab9bcd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240501 20:10:36.385322 7022 raft_consensus.cc:1261] T 024c925714cb419aaa8870a053a6ca2f P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:36.387352 12149 consensus_queue.cc:1035] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:36.390061 4232 raft_consensus.cc:1261] T 024c925714cb419aaa8870a053a6ca2f P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:36.390558 11816 raft_consensus.cc:2932] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:36.391170 11920 consensus_queue.cc:1035] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:36.392071 7022 raft_consensus.cc:2932] T 024c925714cb419aaa8870a053a6ca2f P ef0a4ada548446b3a531cb39a3c4e38d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:36.398695 4232 raft_consensus.cc:2932] T 024c925714cb419aaa8870a053a6ca2f P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } } }
I20240501 20:10:36.399010 4076 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 024c925714cb419aaa8870a053a6ca2f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:36.400945 4091 catalog_manager.cc:5484] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:36.406149 7267 tablet_service.cc:1514] Processing DeleteTablet for tablet 024c925714cb419aaa8870a053a6ca2f with delete_type TABLET_DATA_TOMBSTONED (TS bceeae1c79964acabdae8ff771bf0c97 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54166
@ 0x7f8c39fee8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240501 20:10:36.434082 11146 raft_consensus.cc:1050] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530: attempting to promote NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 to VOTER
I20240501 20:10:36.434931 11146 consensus_queue.cc:237] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:36.447842 11016 raft_consensus.cc:1050] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd: attempting to promote NON_VOTER 10b47782558c41ee91f5e164a372049a to VOTER
I20240501 20:10:36.448452 11016 consensus_queue.cc:237] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } }
I20240501 20:10:36.451236 12161 tablet_replica.cc:330] T 024c925714cb419aaa8870a053a6ca2f P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:36.451546 12161 raft_consensus.cc:2229] T 024c925714cb419aaa8870a053a6ca2f P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:36.451911 12161 raft_consensus.cc:2258] T 024c925714cb419aaa8870a053a6ca2f P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:36.453888 4364 raft_consensus.cc:1261] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.453945 7417 raft_consensus.cc:1261] T 1d3bb8370e7c41dc9cf7f1c835ac275b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.454507 12162 ts_tablet_manager.cc:902] T 3ffe6b742c3743d8b7de7a0c4105637f P 3e974c03d3c546f9ab6c91e762a5426b: Initiating tablet copy from peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:36.454645 11814 consensus_queue.cc:1035] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.455157 4627 raft_consensus.cc:1261] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.455224 11814 consensus_queue.cc:1035] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.455652 11910 consensus_queue.cc:1035] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.467056 12161 ts_tablet_manager.cc:1875] T 024c925714cb419aaa8870a053a6ca2f P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:36.479575 11910 raft_consensus.cc:2932] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:36.480398 7417 raft_consensus.cc:2932] T 1d3bb8370e7c41dc9cf7f1c835ac275b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:36.481329 4365 raft_consensus.cc:2932] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
@ 0x7f8c3a006daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 20:10:36.487241 7281 raft_consensus.cc:1261] T 92ce2c1ce3e0449bab5490b690721cbf P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.487984 12149 consensus_queue.cc:1035] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.490104 4085 catalog_manager.cc:5484] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 2 to 3, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:36.496599 4364 raft_consensus.cc:1261] T 92ce2c1ce3e0449bab5490b690721cbf P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.496973 4627 raft_consensus.cc:1203] T 56b2082919b548bfb56b939373caf770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:36.497201 12149 consensus_queue.cc:1035] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.498262 7419 raft_consensus.cc:1261] T 92ce2c1ce3e0449bab5490b690721cbf P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.501734 11828 raft_consensus.cc:1050] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97: attempting to promote NON_VOTER db0742ee3860468590eefef6ae6074fd to VOTER
I20240501 20:10:36.501523 4627 raft_consensus.cc:2932] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:36.502389 11828 consensus_queue.cc:237] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:36.505173 4499 raft_consensus.cc:1261] T ce0b97a1badb4935a4553902e6feb20b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.505791 7153 raft_consensus.cc:1261] T ce0b97a1badb4935a4553902e6feb20b P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.506234 11935 consensus_queue.cc:1035] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.506749 11935 consensus_queue.cc:1035] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.508314 12149 consensus_queue.cc:1035] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.526360 12162 tablet_copy_client.cc:308] T 3ffe6b742c3743d8b7de7a0c4105637f P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.68:45205
I20240501 20:10:36.527918 4651 tablet_copy_service.cc:140] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Received BeginTabletCopySession request for tablet 3ffe6b742c3743d8b7de7a0c4105637f from peer 3e974c03d3c546f9ab6c91e762a5426b ({username='slave'} at 127.29.219.67:55693)
I20240501 20:10:36.528062 4651 tablet_copy_service.cc:161] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Beginning new tablet copy session on tablet 3ffe6b742c3743d8b7de7a0c4105637f from peer 3e974c03d3c546f9ab6c91e762a5426b at {username='slave'} at 127.29.219.67:55693: session id = 3e974c03d3c546f9ab6c91e762a5426b-3ffe6b742c3743d8b7de7a0c4105637f
I20240501 20:10:36.527976 4232 consensus_queue.cc:237] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:36.532141 4651 tablet_copy_source_session.cc:205] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:36.538415 12162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ffe6b742c3743d8b7de7a0c4105637f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:36.538746 4365 raft_consensus.cc:1261] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:36.538945 7417 raft_consensus.cc:1261] T 1d3bb8370e7c41dc9cf7f1c835ac275b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:36.539388 11814 consensus_queue.cc:1035] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:36.539817 11146 consensus_queue.cc:1035] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:36.542991 12170 raft_consensus.cc:2932] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:36.547466 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1d3bb8370e7c41dc9cf7f1c835ac275b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:36.547353 4365 raft_consensus.cc:2932] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:36.551275 4087 catalog_manager.cc:5484] T 1d3bb8370e7c41dc9cf7f1c835ac275b P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 3 to 4, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:36.552937 12149 raft_consensus.cc:2932] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:36.553733 7417 raft_consensus.cc:2932] T 1d3bb8370e7c41dc9cf7f1c835ac275b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:36.558128 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d3bb8370e7c41dc9cf7f1c835ac275b with delete_type TABLET_DATA_TOMBSTONED (TS 1d34e18a1d394d3f91c9f50d1c7bea4e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51236
I20240501 20:10:36.570257 12182 tablet_replica.cc:330] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 1d34e18a1d394d3f91c9f50d1c7bea4e: stopping tablet replica
I20240501 20:10:36.570462 12182 raft_consensus.cc:2229] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:36.570732 12182 raft_consensus.cc:2258] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:36.570703 4087 catalog_manager.cc:5484] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:36.571424 7419 raft_consensus.cc:2932] T 92ce2c1ce3e0449bab5490b690721cbf P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:36.573419 12182 ts_tablet_manager.cc:1875] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:36.575142 4365 raft_consensus.cc:2932] T 92ce2c1ce3e0449bab5490b690721cbf P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:36.575883 11050 raft_consensus.cc:1050] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b: attempting to promote NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e to VOTER
I20240501 20:10:36.576647 11050 consensus_queue.cc:237] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } }
I20240501 20:10:36.580905 7153 consensus_queue.cc:237] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } }
@ 0x7f8c3a131480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240501 20:10:36.587167 11146 raft_consensus.cc:1050] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530: attempting to promote NON_VOTER db0742ee3860468590eefef6ae6074fd to VOTER
I20240501 20:10:36.588061 11146 consensus_queue.cc:237] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:36.595445 7287 raft_consensus.cc:1261] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.595995 11948 consensus_queue.cc:1035] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.600157 11965 raft_consensus.cc:1050] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d: attempting to promote NON_VOTER dca57823003e4729b769ca12367d7530 to VOTER
I20240501 20:10:36.566231 12181 raft_consensus.cc:2932] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:36.600975 11965 consensus_queue.cc:237] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:36.601751 7281 raft_consensus.cc:2932] T 92ce2c1ce3e0449bab5490b690721cbf P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:36.602715 12161 ts_tablet_manager.cc:1888] T 024c925714cb419aaa8870a053a6ca2f P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:36.602926 12161 log.cc:1196] T 024c925714cb419aaa8870a053a6ca2f P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/024c925714cb419aaa8870a053a6ca2f
I20240501 20:10:36.603163 7419 raft_consensus.cc:1261] T 92ce2c1ce3e0449bab5490b690721cbf P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:36.603960 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet 024c925714cb419aaa8870a053a6ca2f (table rebalancer_extra_table_28 [id=d841e2e5903247b28bb65617a30e22cd]) successfully deleted
I20240501 20:10:36.605880 12149 consensus_queue.cc:1035] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:36.606082 12162 tablet_copy_client.cc:783] T 3ffe6b742c3743d8b7de7a0c4105637f P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:36.606285 12162 tablet_copy_client.cc:647] T 3ffe6b742c3743d8b7de7a0c4105637f P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:36.610706 12149 raft_consensus.cc:2932] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:36.611321 7417 raft_consensus.cc:2932] T 92ce2c1ce3e0449bab5490b690721cbf P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:36.614116 4365 raft_consensus.cc:1261] T 92ce2c1ce3e0449bab5490b690721cbf P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:36.614777 11816 consensus_queue.cc:1035] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:36.615432 12162 tablet_copy_client.cc:523] T 3ffe6b742c3743d8b7de7a0c4105637f P 3e974c03d3c546f9ab6c91e762a5426b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:36.618009 4076 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 92ce2c1ce3e0449bab5490b690721cbf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:36.619827 4365 raft_consensus.cc:2932] T 92ce2c1ce3e0449bab5490b690721cbf P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:36.622342 4085 catalog_manager.cc:5484] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:36.624390 4627 raft_consensus.cc:1261] T 786296114e8a47c68df103934ec055d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
@ 0x7f8c3a169a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240501 20:10:36.624990 11948 consensus_queue.cc:1035] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.626497 12182 ts_tablet_manager.cc:1888] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:36.626598 12182 log.cc:1196] T 1d3bb8370e7c41dc9cf7f1c835ac275b P 1d34e18a1d394d3f91c9f50d1c7bea4e: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/wals/1d3bb8370e7c41dc9cf7f1c835ac275b
I20240501 20:10:36.626981 7267 tablet_service.cc:1514] Processing DeleteTablet for tablet 92ce2c1ce3e0449bab5490b690721cbf with delete_type TABLET_DATA_TOMBSTONED (TS bceeae1c79964acabdae8ff771bf0c97 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54166
I20240501 20:10:36.629887 4079 catalog_manager.cc:4830] TS 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): tablet 1d3bb8370e7c41dc9cf7f1c835ac275b (table rebalancer_extra_table_50 [id=19d9a775c8ed49b4919760e18d418208]) successfully deleted
I20240501 20:10:36.631111 12162 tablet_bootstrap.cc:492] T 3ffe6b742c3743d8b7de7a0c4105637f P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap starting.
I20240501 20:10:36.637039 7417 raft_consensus.cc:1261] T 786296114e8a47c68df103934ec055d5 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.637928 12161 tablet_replica.cc:330] T 92ce2c1ce3e0449bab5490b690721cbf P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:36.638147 12161 raft_consensus.cc:2229] T 92ce2c1ce3e0449bab5490b690721cbf P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:36.638373 12161 raft_consensus.cc:2258] T 92ce2c1ce3e0449bab5490b690721cbf P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:36.639382 12161 ts_tablet_manager.cc:1875] T 92ce2c1ce3e0449bab5490b690721cbf P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:36.640753 11050 consensus_queue.cc:1035] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.641510 7153 raft_consensus.cc:2932] T ce0b97a1badb4935a4553902e6feb20b P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
@ 0x7f8c3a14c981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240501 20:10:36.651626 12161 ts_tablet_manager.cc:1888] T 92ce2c1ce3e0449bab5490b690721cbf P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:36.651791 12161 log.cc:1196] T 92ce2c1ce3e0449bab5490b690721cbf P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/92ce2c1ce3e0449bab5490b690721cbf
I20240501 20:10:36.652513 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet 92ce2c1ce3e0449bab5490b690721cbf (table rebalancer_extra_table_14 [id=be6eb76edc624f598b8543519c4d4476]) successfully deleted
I20240501 20:10:36.652681 11816 raft_consensus.cc:1050] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd: attempting to promote NON_VOTER 10b47782558c41ee91f5e164a372049a to VOTER
I20240501 20:10:36.653421 11816 consensus_queue.cc:237] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } }
I20240501 20:10:36.653996 12105 ts_tablet_manager.cc:902] T 79af181b5f014a40a4c123b47a720dec P 1d34e18a1d394d3f91c9f50d1c7bea4e: Initiating tablet copy from peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:36.657951 4085 catalog_manager.cc:5484] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index 2 to 3, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:36.663758 7153 raft_consensus.cc:1261] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd [term 1 LEARNER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.664240 11814 consensus_queue.cc:1035] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.664897 7419 raft_consensus.cc:1261] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.666958 4363 raft_consensus.cc:1261] T 82260307259e4a9fb12231ff4abd79c5 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
@ 0x7f8c3a14c70e std::_Function_handler<>::_M_invoke() at ??:0
I20240501 20:10:36.667915 11146 consensus_queue.cc:1035] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.671381 11948 raft_consensus.cc:2932] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240501 20:10:36.677775 7280 raft_consensus.cc:2932] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:36.679491 11146 consensus_queue.cc:1035] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.682255 7416 raft_consensus.cc:2932] T 786296114e8a47c68df103934ec055d5 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:36.684520 4628 raft_consensus.cc:2932] T 786296114e8a47c68df103934ec055d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
@ 0x7f8c2aba5b75 kudu::ThreadPool::DispatchThread() at ??:0
I20240501 20:10:36.691358 4364 raft_consensus.cc:1261] T 9709a8bb107742b79078f91da9fab8f6 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.692130 11920 consensus_queue.cc:1035] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.693950 7418 raft_consensus.cc:1261] T 9709a8bb107742b79078f91da9fab8f6 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.695787 4090 catalog_manager.cc:5484] T 786296114e8a47c68df103934ec055d5 P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:36.700573 11920 consensus_queue.cc:1035] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.701849 4232 raft_consensus.cc:1261] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.702426 11840 consensus_queue.cc:1035] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.703004 4367 raft_consensus.cc:1261] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
@ 0x7f8c2aba919c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20240501 20:10:36.707459 11965 consensus_queue.cc:1035] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
@ 0x7f8c2aba9019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f8c2ab8401e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8c2df096db start_thread at ??:0
I20240501 20:10:36.748752 11965 raft_consensus.cc:2932] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:36.740046 7153 raft_consensus.cc:1261] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer ef0a4ada548446b3a531cb39a3c4e38d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.750573 12149 raft_consensus.cc:2932] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:36.905284 11836 consensus_queue.cc:1035] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.913004 4091 catalog_manager.cc:5484] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd reported cstate change: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:36.915848 4085 catalog_manager.cc:5484] T cd21a0faa3bb487cb80f0951b12b3d67 P ef0a4ada548446b3a531cb39a3c4e38d reported cstate change: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:36.807061 4363 raft_consensus.cc:2932] T 82260307259e4a9fb12231ff4abd79c5 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:36.718130 4499 consensus_queue.cc:237] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } }
I20240501 20:10:36.806819 12162 tablet_bootstrap.cc:492] T 3ffe6b742c3743d8b7de7a0c4105637f P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:36.895675 11924 raft_consensus.cc:1050] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b: attempting to promote NON_VOTER bceeae1c79964acabdae8ff771bf0c97 to VOTER
I20240501 20:10:36.926339 12162 tablet_bootstrap.cc:492] T 3ffe6b742c3743d8b7de7a0c4105637f P 3e974c03d3c546f9ab6c91e762a5426b: Bootstrap complete.
I20240501 20:10:36.926640 12162 ts_tablet_manager.cc:1367] T 3ffe6b742c3743d8b7de7a0c4105637f P 3e974c03d3c546f9ab6c91e762a5426b: Time spent bootstrapping tablet: real 0.296s user 0.039s sys 0.028s
I20240501 20:10:36.926908 11924 consensus_queue.cc:237] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:36.927332 12162 raft_consensus.cc:348] T 3ffe6b742c3743d8b7de7a0c4105637f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
I20240501 20:10:36.927517 12162 raft_consensus.cc:729] T 3ffe6b742c3743d8b7de7a0c4105637f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Initialized, Role: LEARNER
I20240501 20:10:36.936309 11965 raft_consensus.cc:1050] T 99bd0e9f4cd7496785ad71480d7d3d61 P ef0a4ada548446b3a531cb39a3c4e38d: attempting to promote NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 to VOTER
I20240501 20:10:36.937523 7153 raft_consensus.cc:2932] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:36.949427 4085 catalog_manager.cc:5484] T 82260307259e4a9fb12231ff4abd79c5 P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index 2 to 3, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
@ 0x7f8c26d4771f clone at ??:0
I20240501 20:10:36.749372 4232 raft_consensus.cc:2932] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:36.764133 11814 raft_consensus.cc:2932] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:36.972803 7155 raft_consensus.cc:2932] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:36.983939 4232 consensus_queue.cc:237] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:36.927690 12162 consensus_queue.cc:260] T 3ffe6b742c3743d8b7de7a0c4105637f P 3e974c03d3c546f9ab6c91e762a5426b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } }
W20240501 20:10:36.824785 7222 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.67:34595, user_credentials={real_user=slave}} blocked reactor thread for 264152us
I20240501 20:10:36.987921 12162 ts_tablet_manager.cc:1398] T 3ffe6b742c3743d8b7de7a0c4105637f P 3e974c03d3c546f9ab6c91e762a5426b: Time spent starting tablet: real 0.061s user 0.001s sys 0.001s
I20240501 20:10:36.933485 7280 raft_consensus.cc:1261] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:36.989838 7155 raft_consensus.cc:1261] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:36.956192 7287 raft_consensus.cc:1261] T 4613582180754f729afc8d0b44356e97 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEARNER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.910053 4364 raft_consensus.cc:2932] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:36.993814 4496 raft_consensus.cc:2932] T ce0b97a1badb4935a4553902e6feb20b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:36.910427 4365 raft_consensus.cc:2932] T 9709a8bb107742b79078f91da9fab8f6 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:36.958761 4363 raft_consensus.cc:1261] T 4613582180754f729afc8d0b44356e97 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.751487 7418 raft_consensus.cc:2932] T 9709a8bb107742b79078f91da9fab8f6 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:36.990831 4368 raft_consensus.cc:1261] T ce0b97a1badb4935a4553902e6feb20b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.765797 7417 raft_consensus.cc:2932] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:36.707317 4627 raft_consensus.cc:1261] T 9709a8bb107742b79078f91da9fab8f6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.841467 12105 tablet_copy_client.cc:308] T 79af181b5f014a40a4c123b47a720dec P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.69:46237
I20240501 20:10:36.770113 12150 raft_consensus.cc:1050] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5: attempting to promote NON_VOTER dca57823003e4729b769ca12367d7530 to VOTER
I20240501 20:10:37.002764 11816 consensus_queue.cc:1035] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:37.003422 12150 consensus_queue.cc:237] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:37.003779 11948 consensus_queue.cc:1035] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:37.004308 11948 consensus_queue.cc:1035] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:36.936353 4628 raft_consensus.cc:1261] T 786296114e8a47c68df103934ec055d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:36.957113 12205 raft_consensus.cc:1050] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5: attempting to promote NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e to VOTER
I20240501 20:10:36.957989 4631 raft_consensus.cc:1261] T 4613582180754f729afc8d0b44356e97 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:36.998134 4651 tablet_copy_service.cc:342] P 1d34e18a1d394d3f91c9f50d1c7bea4e: Request end of tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-3ffe6b742c3743d8b7de7a0c4105637f received from {username='slave'} at 127.29.219.67:55693
I20240501 20:10:37.014631 11935 raft_consensus.cc:1050] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97: attempting to promote NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b to VOTER
I20240501 20:10:37.074128 4651 tablet_copy_service.cc:433] P 1d34e18a1d394d3f91c9f50d1c7bea4e: ending tablet copy session 3e974c03d3c546f9ab6c91e762a5426b-3ffe6b742c3743d8b7de7a0c4105637f on tablet 3ffe6b742c3743d8b7de7a0c4105637f with peer 3e974c03d3c546f9ab6c91e762a5426b
I20240501 20:10:37.060292 4628 raft_consensus.cc:2932] T 9709a8bb107742b79078f91da9fab8f6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:37.075089 11935 consensus_queue.cc:237] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
W20240501 20:10:37.059091 7222 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.65:42941, user_credentials={real_user=slave}} blocked reactor thread for 44992.8us
I20240501 20:10:37.080426 11828 consensus_queue.cc:1035] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 20:10:37.062314 4300 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.71:39447, user_credentials={real_user=slave}} blocked reactor thread for 56924.5us
I20240501 20:10:37.006433 12205 consensus_queue.cc:237] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } }
I20240501 20:10:37.089963 7419 raft_consensus.cc:1261] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:37.012154 11050 consensus_queue.cc:1035] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:37.090723 7417 raft_consensus.cc:1261] T 376ae6034f394a0fb0cab0d8a9241134 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:37.012461 11948 consensus_queue.cc:1035] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:37.038580 4496 raft_consensus.cc:1203] T 3ffe6b742c3743d8b7de7a0c4105637f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240501 20:10:37.060813 4433 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.71:39447, user_credentials={real_user=slave}} blocked reactor thread for 43893us
I20240501 20:10:37.064353 4499 raft_consensus.cc:1261] T 9c77905da1b3453fa9a83366f7c51c37 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:37.088852 4498 raft_consensus.cc:1261] T 376ae6034f394a0fb0cab0d8a9241134 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:37.178640 12150 consensus_queue.cc:1035] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:37.179862 11935 consensus_queue.cc:1035] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:37.184193 11924 consensus_queue.cc:1035] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:37.187238 4628 raft_consensus.cc:1261] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:37.189837 7286 raft_consensus.cc:1261] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:37.185484 12241 raft_consensus.cc:2932] T 786296114e8a47c68df103934ec055d5 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:37.189680 7284 raft_consensus.cc:2932] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:37.192324 4368 raft_consensus.cc:2932] T ce0b97a1badb4935a4553902e6feb20b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, db0742ee3860468590eefef6ae6074fd (127.29.219.70) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:37.193051 4631 raft_consensus.cc:2932] T 4613582180754f729afc8d0b44356e97 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:37.194867 11828 consensus_queue.cc:1035] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:37.191864 12240 raft_consensus.cc:2932] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:37.011250 11814 consensus_queue.cc:1035] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:37.193689 4627 raft_consensus.cc:2932] T 786296114e8a47c68df103934ec055d5 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:37.217206 4366 raft_consensus.cc:2932] T 4613582180754f729afc8d0b44356e97 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:37.060134 4232 raft_consensus.cc:1261] T 9c77905da1b3453fa9a83366f7c51c37 P dca57823003e4729b769ca12367d7530 [term 1 LEARNER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:37.087900 4233 raft_consensus.cc:1261] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:37.257156 11910 consensus_queue.cc:1035] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:37.260195 4085 catalog_manager.cc:5484] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:37.262387 7284 raft_consensus.cc:2932] T 4613582180754f729afc8d0b44356e97 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:37.260535 4091 catalog_manager.cc:5484] T 4613582180754f729afc8d0b44356e97 P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index 2 to 3, bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:37.263715 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 786296114e8a47c68df103934ec055d5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:37.187825 12150 consensus_queue.cc:1035] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:37.269565 12014 consensus_queue.cc:1035] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:37.268445 11935 consensus_queue.cc:1035] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:37.271603 7400 tablet_service.cc:1514] Processing DeleteTablet for tablet 786296114e8a47c68df103934ec055d5 with delete_type TABLET_DATA_TOMBSTONED (TS b9957d4086bc4a76b18ed9265a5739d5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40182
I20240501 20:10:37.274773 7419 raft_consensus.cc:2932] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:37.285415 12205 consensus_queue.cc:1035] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:37.288194 4498 raft_consensus.cc:1261] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:37.289345 12244 consensus_queue.cc:1035] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:37.291878 4499 raft_consensus.cc:2932] T 376ae6034f394a0fb0cab0d8a9241134 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:37.302398 4496 consensus_queue.cc:237] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:37.303947 12268 tablet_replica.cc:330] T 786296114e8a47c68df103934ec055d5 P b9957d4086bc4a76b18ed9265a5739d5: stopping tablet replica
I20240501 20:10:37.262307 11910 raft_consensus.cc:2932] T 82260307259e4a9fb12231ff4abd79c5 P dca57823003e4729b769ca12367d7530 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:37.304736 4631 raft_consensus.cc:1261] T 4613582180754f729afc8d0b44356e97 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:37.304908 12268 raft_consensus.cc:2229] T 786296114e8a47c68df103934ec055d5 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:37.305294 12241 consensus_queue.cc:1035] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:37.305308 12268 raft_consensus.cc:2258] T 786296114e8a47c68df103934ec055d5 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:37.307708 7286 raft_consensus.cc:1261] T 4613582180754f729afc8d0b44356e97 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:37.308384 12241 raft_consensus.cc:2932] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:37.287917 11828 raft_consensus.cc:2932] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:37.309079 4628 raft_consensus.cc:2932] T 4613582180754f729afc8d0b44356e97 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:37.316473 12240 consensus_queue.cc:1035] T 4613582180754f729afc8d0b44356e97 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:37.318143 7286 raft_consensus.cc:2932] T 4613582180754f729afc8d0b44356e97 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:37.318279 4090 catalog_manager.cc:5484] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: config changed from index 3 to 4, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New cstate: current_term: 1 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:37.266182 7155 raft_consensus.cc:2932] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:37.322185 4087 catalog_manager.cc:5484] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:37.323803 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4613582180754f729afc8d0b44356e97 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:37.326817 4091 catalog_manager.cc:5484] T 4613582180754f729afc8d0b44356e97 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 3 to 4, VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) evicted. New cstate: current_term: 1 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } }
I20240501 20:10:37.328052 7420 raft_consensus.cc:2932] T 376ae6034f394a0fb0cab0d8a9241134 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:37.333266 7286 consensus_queue.cc:237] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:37.334872 4346 tablet_service.cc:1514] Processing DeleteTablet for tablet 4613582180754f729afc8d0b44356e97 with delete_type TABLET_DATA_TOMBSTONED (TS 10b47782558c41ee91f5e164a372049a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:37.337409 4348 tablet_service.cc:1514] Processing DeleteTablet for tablet 82260307259e4a9fb12231ff4abd79c5 with delete_type TABLET_DATA_TOMBSTONED (TS 10b47782558c41ee91f5e164a372049a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60928
I20240501 20:10:37.338236 7420 raft_consensus.cc:1261] T 376ae6034f394a0fb0cab0d8a9241134 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:37.339037 12239 consensus_queue.cc:1035] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:37.339785 12268 ts_tablet_manager.cc:1875] T 786296114e8a47c68df103934ec055d5 P b9957d4086bc4a76b18ed9265a5739d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:37.345927 4495 raft_consensus.cc:1261] T 376ae6034f394a0fb0cab0d8a9241134 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:37.347170 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 82260307259e4a9fb12231ff4abd79c5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:37.348184 11828 consensus_queue.cc:1035] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:37.353159 12244 raft_consensus.cc:2932] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:37.354923 4627 raft_consensus.cc:2932] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:37.359377 7284 raft_consensus.cc:2932] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:37.359373 4498 raft_consensus.cc:2932] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:37.367594 12183 raft_consensus.cc:2932] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:37.368775 7419 raft_consensus.cc:2932] T 376ae6034f394a0fb0cab0d8a9241134 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:37.369901 12268 ts_tablet_manager.cc:1888] T 786296114e8a47c68df103934ec055d5 P b9957d4086bc4a76b18ed9265a5739d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:37.371569 12268 log.cc:1196] T 786296114e8a47c68df103934ec055d5 P b9957d4086bc4a76b18ed9265a5739d5: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/wals/786296114e8a47c68df103934ec055d5
I20240501 20:10:37.374307 4077 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 376ae6034f394a0fb0cab0d8a9241134 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:37.376195 4079 catalog_manager.cc:4830] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): tablet 786296114e8a47c68df103934ec055d5 (table rebalancer_extra_table_46 [id=b8cce50d935840afb7fe769c351116d9]) successfully deleted
I20240501 20:10:37.377168 4233 raft_consensus.cc:2932] T 9c77905da1b3453fa9a83366f7c51c37 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:37.381187 12205 raft_consensus.cc:2932] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:37.384287 4090 catalog_manager.cc:5484] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 2 to 3, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:37.390121 4496 raft_consensus.cc:2932] T 9c77905da1b3453fa9a83366f7c51c37 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:37.401578 4091 catalog_manager.cc:5484] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index 3 to 4, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:37.409804 4495 raft_consensus.cc:2932] T 376ae6034f394a0fb0cab0d8a9241134 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dca57823003e4729b769ca12367d7530 (127.29.219.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:37.414587 4215 tablet_service.cc:1514] Processing DeleteTablet for tablet 376ae6034f394a0fb0cab0d8a9241134 with delete_type TABLET_DATA_TOMBSTONED (TS dca57823003e4729b769ca12367d7530 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58624
I20240501 20:10:37.415338 12275 tablet_replica.cc:330] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530: stopping tablet replica
I20240501 20:10:37.415580 12275 raft_consensus.cc:2229] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:37.415791 12275 pending_rounds.cc:62] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530: Trying to abort 1 pending ops.
I20240501 20:10:37.415841 12275 pending_rounds.cc:68] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7022977995056803840 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "376ae6034f394a0fb0cab0d8a9241134" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: NON_VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } } }
I20240501 20:10:37.416232 12275 raft_consensus.cc:2894] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240501 20:10:37.416335 12275 raft_consensus.cc:2866] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 20:10:37.416579 12275 raft_consensus.cc:2258] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:37.418573 12274 tablet_replica.cc:330] T 4613582180754f729afc8d0b44356e97 P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:37.418967 12274 raft_consensus.cc:2229] T 4613582180754f729afc8d0b44356e97 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:37.419234 12274 raft_consensus.cc:2258] T 4613582180754f729afc8d0b44356e97 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:37.420356 12275 ts_tablet_manager.cc:1875] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:37.420766 7419 consensus_queue.cc:237] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } }
I20240501 20:10:37.423095 4085 catalog_manager.cc:5484] T 9c77905da1b3453fa9a83366f7c51c37 P dca57823003e4729b769ca12367d7530 reported cstate change: config changed from index 2 to 3, dca57823003e4729b769ca12367d7530 (127.29.219.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } } }
I20240501 20:10:37.425850 4495 raft_consensus.cc:1261] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:37.426445 12205 consensus_queue.cc:1035] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:37.427102 12275 ts_tablet_manager.cc:1888] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:37.427203 12275 log.cc:1196] T 376ae6034f394a0fb0cab0d8a9241134 P dca57823003e4729b769ca12367d7530: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/wals/376ae6034f394a0fb0cab0d8a9241134
I20240501 20:10:37.428313 4627 raft_consensus.cc:1261] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:37.428951 12205 consensus_queue.cc:1035] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:37.429621 4078 catalog_manager.cc:4830] TS dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): tablet 376ae6034f394a0fb0cab0d8a9241134 (table rebalancer_extra_table_42 [id=0a6696a672f449e89bcc33228cf8c494]) successfully deleted
I20240501 20:10:37.429342 12150 raft_consensus.cc:2932] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:37.434051 4495 raft_consensus.cc:2932] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:37.434885 4079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9e0ecc1043af4bc1ab31dfa80ea47bb4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:37.434947 4627 raft_consensus.cc:2932] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:37.441448 12274 ts_tablet_manager.cc:1875] T 4613582180754f729afc8d0b44356e97 P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:37.444420 4085 catalog_manager.cc:5484] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 3 to 4, VOTER bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71) evicted. New cstate: current_term: 1 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:37.446056 12274 ts_tablet_manager.cc:1888] T 4613582180754f729afc8d0b44356e97 P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:37.446167 12274 log.cc:1196] T 4613582180754f729afc8d0b44356e97 P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/4613582180754f729afc8d0b44356e97
I20240501 20:10:37.446843 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet 4613582180754f729afc8d0b44356e97 (table rebalancer_extra_table_40 [id=c333ed244e1f4842956fe44221ea0c02]) successfully deleted
I20240501 20:10:37.449050 12274 tablet_replica.cc:330] T 82260307259e4a9fb12231ff4abd79c5 P 10b47782558c41ee91f5e164a372049a: stopping tablet replica
I20240501 20:10:37.450796 12274 raft_consensus.cc:2229] T 82260307259e4a9fb12231ff4abd79c5 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:37.452848 7267 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e0ecc1043af4bc1ab31dfa80ea47bb4 with delete_type TABLET_DATA_TOMBSTONED (TS bceeae1c79964acabdae8ff771bf0c97 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54166
I20240501 20:10:37.453734 12276 tablet_replica.cc:330] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P bceeae1c79964acabdae8ff771bf0c97: stopping tablet replica
I20240501 20:10:37.453986 12276 raft_consensus.cc:2229] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:37.454306 12276 raft_consensus.cc:2258] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 20:10:37.454294 4564 connection.cc:535] client connection to 127.29.219.69:46237 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:37.454294 7355 connection.cc:535] client connection to 127.29.219.69:46237 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:37.454524 4564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:37.454562 7355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:37.455442 12276 ts_tablet_manager.cc:1875] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P bceeae1c79964acabdae8ff771bf0c97: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:37.455439 12274 raft_consensus.cc:2258] T 82260307259e4a9fb12231ff4abd79c5 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:37.456497 12274 ts_tablet_manager.cc:1875] T 82260307259e4a9fb12231ff4abd79c5 P 10b47782558c41ee91f5e164a372049a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 20:10:37.460366 4434 connection.cc:535] client connection to 127.29.219.69:46237 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:37.460649 4434 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:37.462145 12276 ts_tablet_manager.cc:1888] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P bceeae1c79964acabdae8ff771bf0c97: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:37.462332 12276 log.cc:1196] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P bceeae1c79964acabdae8ff771bf0c97: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/wals/9e0ecc1043af4bc1ab31dfa80ea47bb4
I20240501 20:10:37.463186 4077 catalog_manager.cc:4830] TS bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): tablet 9e0ecc1043af4bc1ab31dfa80ea47bb4 (table rebalancer_extra_table_48 [id=4a624bfe1b6b4d6b8ab2c2c578c11688]) successfully deleted
I20240501 20:10:37.465694 12274 ts_tablet_manager.cc:1888] T 82260307259e4a9fb12231ff4abd79c5 P 10b47782558c41ee91f5e164a372049a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:37.465850 12274 log.cc:1196] T 82260307259e4a9fb12231ff4abd79c5 P 10b47782558c41ee91f5e164a372049a: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/wals/82260307259e4a9fb12231ff4abd79c5
I20240501 20:10:37.466526 4078 catalog_manager.cc:4830] TS 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): tablet 82260307259e4a9fb12231ff4abd79c5 (table rebalancer_extra_table_40 [id=c333ed244e1f4842956fe44221ea0c02]) successfully deleted
W20240501 20:10:37.470283 4169 connection.cc:535] server connection from 127.29.219.69:51289 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:37.470492 4169 connection.cc:535] client connection to 127.29.219.69:46237 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:37.470746 4169 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:37.470820 4302 connection.cc:535] server connection from 127.29.219.69:41755 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:37.470849 7090 connection.cc:535] server connection from 127.29.219.69:39809 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:37.471143 4565 connection.cc:535] server connection from 127.29.219.69:59565 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:37.472981 4301 connection.cc:535] client connection to 127.29.219.69:46237 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:37.473237 4301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:37.477661 7090 connection.cc:535] client connection to 127.29.219.69:46237 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:37.477720 7222 connection.cc:535] client connection to 127.29.219.69:46237 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:37.477903 7090 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:37.477952 7222 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:37.479799 4565 connection.cc:441] server connection from 127.29.219.69:59565 torn down before Call kudu.consensus.ConsensusService.StartTabletCopy from 127.29.219.69:59565 (request call id 288) could send its response
W20240501 20:10:37.482100 12282 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:10:37.472190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.219.69:46237 (local address 127.29.219.67:35179)
0501 20:10:37.481054 (+ 8864us) negotiation.cc:107] Waiting for socket to connect
0501 20:10:37.481225 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":8729,"threads_started":1}
W20240501 20:10:37.485895 12281 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:10:37.471528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.219.69:46237 (local address 127.29.219.72:49665)
0501 20:10:37.484822 (+ 13294us) negotiation.cc:107] Waiting for socket to connect
0501 20:10:37.485014 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":13151,"thread_start_us":13532,"threads_started":1}
W20240501 20:10:37.486387 4434 consensus_peers.cc:485] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.486428 7222 consensus_peers.cc:485] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.487095 4434 consensus_peers.cc:485] T 09dcd419921941e9a19cfd9a20b89c70 P 3e974c03d3c546f9ab6c91e762a5426b -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.487385 4434 consensus_peers.cc:485] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.487617 4434 consensus_peers.cc:485] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.488157 7355 consensus_peers.cc:485] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.488842 7355 consensus_peers.cc:485] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.489073 7355 consensus_peers.cc:485] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.492462 4564 consensus_peers.cc:485] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.492671 4564 consensus_peers.cc:485] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.492869 4564 consensus_peers.cc:485] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.493036 12283 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:10:37.472673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.219.69:46237 (local address 127.29.219.65:56339)
0501 20:10:37.492574 (+ 19901us) negotiation.cc:107] Waiting for socket to connect
0501 20:10:37.492723 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":19757,"thread_start_us":19674,"threads_started":1}
W20240501 20:10:37.493039 4564 consensus_peers.cc:485] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.493221 4564 consensus_peers.cc:485] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.493312 4169 consensus_peers.cc:485] T c9025370d3f44514a1430375e27c0239 P dca57823003e4729b769ca12367d7530 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.493424 4169 consensus_peers.cc:485] T 9d634c1feb7449f686cf3af2a07181e0 P dca57823003e4729b769ca12367d7530 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.493511 4169 consensus_peers.cc:485] T f4f03f407a364b80b1925418d0bdbc5d P dca57823003e4729b769ca12367d7530 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.493583 4169 consensus_peers.cc:485] T 644d4d293e8c462297c6419965bd2c89 P dca57823003e4729b769ca12367d7530 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.495633 7222 consensus_peers.cc:485] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.496850 7222 consensus_peers.cc:485] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.496973 7222 consensus_peers.cc:485] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.497097 7090 consensus_peers.cc:485] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.497225 7090 consensus_peers.cc:462] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
W20240501 20:10:37.497329 7090 consensus_peers.cc:485] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.497426 7090 consensus_peers.cc:485] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.497517 7090 consensus_peers.cc:485] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.497622 7090 consensus_peers.cc:485] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.497730 7090 consensus_peers.cc:485] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.499986 4301 consensus_peers.cc:485] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.500138 4301 consensus_peers.cc:485] T 3a982d9467a14f88a54a240266be728d P 10b47782558c41ee91f5e164a372049a -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:37.526932 7090 consensus_peers.cc:485] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:37.532917 12267 raft_consensus.cc:1050] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e: attempting to promote NON_VOTER 3e974c03d3c546f9ab6c91e762a5426b to VOTER
I20240501 20:10:37.533668 12267 consensus_queue.cc:237] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:37.540038 4495 raft_consensus.cc:1261] T 3ffe6b742c3743d8b7de7a0c4105637f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 LEARNER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:37.540853 11999 consensus_queue.cc:1035] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:37.541787 4233 raft_consensus.cc:1261] T 3ffe6b742c3743d8b7de7a0c4105637f P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:37.543896 11999 consensus_queue.cc:1035] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:37.548020 7153 raft_consensus.cc:1261] T 3ffe6b742c3743d8b7de7a0c4105637f P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:37.548688 11999 consensus_queue.cc:1035] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:37.556025 11999 raft_consensus.cc:2932] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:37.561007 7153 raft_consensus.cc:2932] T 3ffe6b742c3743d8b7de7a0c4105637f P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:37.561753 4495 raft_consensus.cc:2932] T 3ffe6b742c3743d8b7de7a0c4105637f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:37.569002 4087 catalog_manager.cc:5484] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:37.570609 4233 raft_consensus.cc:2932] T 3ffe6b742c3743d8b7de7a0c4105637f P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:37.588129 4627 consensus_queue.cc:237] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:37.593055 4495 raft_consensus.cc:1261] T 3ffe6b742c3743d8b7de7a0c4105637f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:37.593650 4233 raft_consensus.cc:1261] T 3ffe6b742c3743d8b7de7a0c4105637f P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:37.594000 12119 consensus_queue.cc:1035] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:37.597354 11999 consensus_queue.cc:1035] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:37.600880 12267 raft_consensus.cc:2932] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:37.605852 4079 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3ffe6b742c3743d8b7de7a0c4105637f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:37.606536 4498 raft_consensus.cc:2932] T 3ffe6b742c3743d8b7de7a0c4105637f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:37.613648 4085 catalog_manager.cc:5484] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New cstate: current_term: 1 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:37.617648 4233 raft_consensus.cc:2932] T 3ffe6b742c3743d8b7de7a0c4105637f P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER db0742ee3860468590eefef6ae6074fd (127.29.219.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } }
I20240501 20:10:37.618443 7286 consensus_queue.cc:237] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:37.620920 4368 raft_consensus.cc:1261] T ce0b97a1badb4935a4553902e6feb20b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:37.620970 7153 raft_consensus.cc:1261] T ce0b97a1badb4935a4553902e6feb20b P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:37.621847 12183 consensus_queue.cc:1035] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:37.623188 12183 consensus_queue.cc:1035] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:37.626571 11828 raft_consensus.cc:2932] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:37.627010 7135 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ffe6b742c3743d8b7de7a0c4105637f with delete_type TABLET_DATA_TOMBSTONED (TS db0742ee3860468590eefef6ae6074fd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52036
I20240501 20:10:37.632040 4368 raft_consensus.cc:2932] T ce0b97a1badb4935a4553902e6feb20b P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:37.632287 7153 raft_consensus.cc:2932] T ce0b97a1badb4935a4553902e6feb20b P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } } }
I20240501 20:10:37.640650 7155 consensus_queue.cc:237] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } }
I20240501 20:10:37.643051 4631 raft_consensus.cc:1261] T 9709a8bb107742b79078f91da9fab8f6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:37.645100 4077 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ce0b97a1badb4935a4553902e6feb20b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:37.648373 11920 consensus_queue.cc:1035] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:37.650499 4368 raft_consensus.cc:1261] T 9709a8bb107742b79078f91da9fab8f6 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:37.652096 4085 catalog_manager.cc:5484] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index 3 to 4, VOTER 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67) evicted. New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:37.654037 11920 consensus_queue.cc:1035] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:37.656822 11920 raft_consensus.cc:2932] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:37.661338 4366 raft_consensus.cc:2932] T 9709a8bb107742b79078f91da9fab8f6 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:37.662343 4478 tablet_service.cc:1514] Processing DeleteTablet for tablet ce0b97a1badb4935a4553902e6feb20b with delete_type TABLET_DATA_TOMBSTONED (TS 3e974c03d3c546f9ab6c91e762a5426b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55490
I20240501 20:10:37.665145 12329 tablet_replica.cc:330] T 3ffe6b742c3743d8b7de7a0c4105637f P db0742ee3860468590eefef6ae6074fd: stopping tablet replica
I20240501 20:10:37.665357 12329 raft_consensus.cc:2229] T 3ffe6b742c3743d8b7de7a0c4105637f P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:37.666237 12329 raft_consensus.cc:2258] T 3ffe6b742c3743d8b7de7a0c4105637f P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:37.667194 12329 ts_tablet_manager.cc:1875] T 3ffe6b742c3743d8b7de7a0c4105637f P db0742ee3860468590eefef6ae6074fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:37.669217 4085 catalog_manager.cc:5484] T 9709a8bb107742b79078f91da9fab8f6 P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New cstate: current_term: 1 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:37.669202 4627 raft_consensus.cc:2932] T 9709a8bb107742b79078f91da9fab8f6 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:37.672941 4076 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9709a8bb107742b79078f91da9fab8f6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:37.673732 7400 tablet_service.cc:1514] Processing DeleteTablet for tablet 9709a8bb107742b79078f91da9fab8f6 with delete_type TABLET_DATA_TOMBSTONED (TS b9957d4086bc4a76b18ed9265a5739d5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40182
I20240501 20:10:37.674154 12268 tablet_replica.cc:330] T 9709a8bb107742b79078f91da9fab8f6 P b9957d4086bc4a76b18ed9265a5739d5: stopping tablet replica
I20240501 20:10:37.674360 12268 raft_consensus.cc:2229] T 9709a8bb107742b79078f91da9fab8f6 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:37.674688 12268 raft_consensus.cc:2258] T 9709a8bb107742b79078f91da9fab8f6 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:37.675506 12268 ts_tablet_manager.cc:1875] T 9709a8bb107742b79078f91da9fab8f6 P b9957d4086bc4a76b18ed9265a5739d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:37.678448 12329 ts_tablet_manager.cc:1888] T 3ffe6b742c3743d8b7de7a0c4105637f P db0742ee3860468590eefef6ae6074fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:37.678575 12329 log.cc:1196] T 3ffe6b742c3743d8b7de7a0c4105637f P db0742ee3860468590eefef6ae6074fd: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/wals/3ffe6b742c3743d8b7de7a0c4105637f
I20240501 20:10:37.679610 4076 catalog_manager.cc:4830] TS db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): tablet 3ffe6b742c3743d8b7de7a0c4105637f (table rebalancer_extra_table_52 [id=4966f0b362754500952a1e7f97c65b0b]) successfully deleted
I20240501 20:10:37.681100 12268 ts_tablet_manager.cc:1888] T 9709a8bb107742b79078f91da9fab8f6 P b9957d4086bc4a76b18ed9265a5739d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:37.681232 12268 log.cc:1196] T 9709a8bb107742b79078f91da9fab8f6 P b9957d4086bc4a76b18ed9265a5739d5: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/wals/9709a8bb107742b79078f91da9fab8f6
I20240501 20:10:37.682055 12330 tablet_replica.cc:330] T ce0b97a1badb4935a4553902e6feb20b P 3e974c03d3c546f9ab6c91e762a5426b: stopping tablet replica
I20240501 20:10:37.682253 12330 raft_consensus.cc:2229] T ce0b97a1badb4935a4553902e6feb20b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:10:37.682560 12330 raft_consensus.cc:2258] T ce0b97a1badb4935a4553902e6feb20b P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:10:37.683938 12330 ts_tablet_manager.cc:1875] T ce0b97a1badb4935a4553902e6feb20b P 3e974c03d3c546f9ab6c91e762a5426b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:10:37.684299 4079 catalog_manager.cc:4830] TS b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817): tablet 9709a8bb107742b79078f91da9fab8f6 (table rebalancer_extra_table_42 [id=0a6696a672f449e89bcc33228cf8c494]) successfully deleted
I20240501 20:10:37.700665 12330 ts_tablet_manager.cc:1888] T ce0b97a1badb4935a4553902e6feb20b P 3e974c03d3c546f9ab6c91e762a5426b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 20:10:37.700790 12330 log.cc:1196] T ce0b97a1badb4935a4553902e6feb20b P 3e974c03d3c546f9ab6c91e762a5426b: Deleting WAL directory at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/wals/ce0b97a1badb4935a4553902e6feb20b
I20240501 20:10:37.701560 4078 catalog_manager.cc:4830] TS 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): tablet ce0b97a1badb4935a4553902e6feb20b (table rebalancer_extra_table_38 [id=ccbf7dce50cd4324a764233997f8c643]) successfully deleted
W20240501 20:10:37.792960 7090 consensus_peers.cc:462] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:37.853850 4366 consensus_queue.cc:237] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:37.856405 4627 raft_consensus.cc:1261] T bec3cae7dc9241fd8eae761967932f6e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:37.856428 7419 raft_consensus.cc:1261] T bec3cae7dc9241fd8eae761967932f6e P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:37.857029 12243 consensus_queue.cc:1035] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:37.857329 4301 consensus_peers.cc:485] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:37.857347 12235 consensus_queue.cc:1035] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:37.859831 12243 raft_consensus.cc:2932] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:37.860442 4627 raft_consensus.cc:2932] T bec3cae7dc9241fd8eae761967932f6e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:37.860486 7419 raft_consensus.cc:2932] T bec3cae7dc9241fd8eae761967932f6e P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } } }
I20240501 20:10:37.867142 4091 catalog_manager.cc:5484] T bec3cae7dc9241fd8eae761967932f6e P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index -1 to 2, NON_VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) added. New cstate: current_term: 1 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:37.882925 7286 consensus_queue.cc:237] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:37.885403 4495 raft_consensus.cc:1261] T aecca0e22b084a8b95c98ef7e7b90b29 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 20:10:37.885634 7222 consensus_peers.cc:485] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aecca0e22b084a8b95c98ef7e7b90b29. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:37.886237 11828 consensus_queue.cc:1035] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:37.886725 7222 consensus_peers.cc:485] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:37.888824 11828 raft_consensus.cc:2932] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:37.890393 4498 raft_consensus.cc:2932] T aecca0e22b084a8b95c98ef7e7b90b29 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } } }
I20240501 20:10:37.896859 4085 catalog_manager.cc:5484] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index -1 to 2, NON_VOTER 10b47782558c41ee91f5e164a372049a (127.29.219.66) added. New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:37.945093 11948 raft_consensus.cc:482] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:37.945256 11948 raft_consensus.cc:504] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:37.945741 11948 leader_election.cc:288] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:37.946414 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f18af954cefa48d4882313fe18abbd27" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
W20240501 20:10:37.947098 4434 leader_election.cc:334] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:37.947190 4434 leader_election.cc:302] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b; no voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:37.947427 11948 raft_consensus.cc:2726] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:10:38.004993 12149 raft_consensus.cc:482] T 56b2082919b548bfb56b939373caf770 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.005221 12149 raft_consensus.cc:504] T 56b2082919b548bfb56b939373caf770 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:38.005935 12149 leader_election.cc:288] T 56b2082919b548bfb56b939373caf770 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:38.006884 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b2082919b548bfb56b939373caf770" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
W20240501 20:10:38.010475 7090 leader_election.cc:334] T 56b2082919b548bfb56b939373caf770 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.010650 7090 leader_election.cc:302] T 56b2082919b548bfb56b939373caf770 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db0742ee3860468590eefef6ae6074fd; no voters: 3e974c03d3c546f9ab6c91e762a5426b, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.010896 12149 raft_consensus.cc:2726] T 56b2082919b548bfb56b939373caf770 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:38.026772 11999 raft_consensus.cc:482] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.026962 11999 raft_consensus.cc:504] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:38.027644 11999 leader_election.cc:288] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:38.032500 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f18af954cefa48d4882313fe18abbd27" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:38.032778 4498 raft_consensus.cc:2454] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 2.
I20240501 20:10:38.033947 4564 leader_election.cc:302] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:10:38.034806 12119 raft_consensus.cc:2781] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:10:38.034884 12119 raft_consensus.cc:482] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Starting leader election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.034955 12119 raft_consensus.cc:3037] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:10:38.037297 12119 raft_consensus.cc:504] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:38.038178 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f18af954cefa48d4882313fe18abbd27" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:10:38.038352 4498 raft_consensus.cc:3037] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:10:38.038321 11910 raft_consensus.cc:482] T 12f13fda83114ec5b9dcd0fa90ddef7e P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.038480 11910 raft_consensus.cc:504] T 12f13fda83114ec5b9dcd0fa90ddef7e P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:38.039135 11910 leader_election.cc:288] T 12f13fda83114ec5b9dcd0fa90ddef7e P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:38.041020 4498 raft_consensus.cc:2454] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 3.
I20240501 20:10:38.043285 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f13fda83114ec5b9dcd0fa90ddef7e" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:38.037828 12119 leader_election.cc:288] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 3 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
W20240501 20:10:38.042780 4564 leader_election.cc:334] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.047214 4564 leader_election.cc:302] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
W20240501 20:10:38.047870 4169 leader_election.cc:334] T 12f13fda83114ec5b9dcd0fa90ddef7e P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.048013 11999 raft_consensus.cc:2781] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:10:38.048004 4169 leader_election.cc:302] T 12f13fda83114ec5b9dcd0fa90ddef7e P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dca57823003e4729b769ca12367d7530; no voters: b9957d4086bc4a76b18ed9265a5739d5, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.048115 11999 raft_consensus.cc:686] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 3 LEADER]: Becoming Leader. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Running, Role: LEADER
I20240501 20:10:38.048345 11999 consensus_queue.cc:237] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:38.049281 11146 raft_consensus.cc:2726] T 12f13fda83114ec5b9dcd0fa90ddef7e P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:10:38.049672 4564 leader_election.cc:334] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.053344 4087 catalog_manager.cc:5484] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: term changed from 2 to 3, leader changed from ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) to 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68). New cstate: current_term: 3 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:38.068711 11948 raft_consensus.cc:482] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.068924 11948 raft_consensus.cc:504] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:38.069602 11948 leader_election.cc:288] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:38.070835 4366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79af181b5f014a40a4c123b47a720dec" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
W20240501 20:10:38.071720 4434 leader_election.cc:334] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.071820 4434 leader_election.cc:302] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b; no voters: 10b47782558c41ee91f5e164a372049a, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.072278 11948 raft_consensus.cc:2726] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:38.093858 11146 raft_consensus.cc:482] T accaff8513134f96b5502b5c5385d892 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.094046 11146 raft_consensus.cc:504] T accaff8513134f96b5502b5c5385d892 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:38.094802 11146 leader_election.cc:288] T accaff8513134f96b5502b5c5385d892 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:38.096202 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "accaff8513134f96b5502b5c5385d892" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
W20240501 20:10:38.099049 4169 leader_election.cc:334] T accaff8513134f96b5502b5c5385d892 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.099197 4169 leader_election.cc:302] T accaff8513134f96b5502b5c5385d892 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dca57823003e4729b769ca12367d7530; no voters: b9957d4086bc4a76b18ed9265a5739d5, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.099673 11146 raft_consensus.cc:2726] T accaff8513134f96b5502b5c5385d892 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:38.107966 11146 raft_consensus.cc:482] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.108155 11146 raft_consensus.cc:504] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:38.109138 11146 leader_election.cc:288] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:38.109300 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d860bb204134f0994462275519feecf" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
W20240501 20:10:38.113781 4169 leader_election.cc:334] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.113924 4169 leader_election.cc:302] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dca57823003e4729b769ca12367d7530; no voters: 3e974c03d3c546f9ab6c91e762a5426b, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.114161 11146 raft_consensus.cc:2726] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:38.171085 11948 raft_consensus.cc:482] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.171260 11948 raft_consensus.cc:504] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:38.171905 11948 leader_election.cc:288] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:38.172590 12149 raft_consensus.cc:482] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.172757 12149 raft_consensus.cc:504] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:38.173245 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f63c61d85524d87a4b911c321f486c4" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:38.173435 12149 leader_election.cc:288] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:38.177448 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48528ec242974de4ac20eb0161f0cb05" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
W20240501 20:10:38.180277 7090 leader_election.cc:334] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.180666 7090 leader_election.cc:302] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db0742ee3860468590eefef6ae6074fd; no voters: 3e974c03d3c546f9ab6c91e762a5426b, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.181200 11016 raft_consensus.cc:2726] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:10:38.181926 4434 leader_election.cc:334] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.182029 4434 leader_election.cc:302] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b; no voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.182718 12235 raft_consensus.cc:482] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.182898 12235 raft_consensus.cc:504] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:38.183727 12235 leader_election.cc:288] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:38.185566 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79af181b5f014a40a4c123b47a720dec" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:38.185746 4498 raft_consensus.cc:2454] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:10:38.188413 4301 leader_election.cc:302] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
W20240501 20:10:38.188678 4301 leader_election.cc:334] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.189342 12243 raft_consensus.cc:2781] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:10:38.189433 12243 raft_consensus.cc:482] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.189517 12243 raft_consensus.cc:3037] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:38.192844 11948 raft_consensus.cc:2726] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:38.194890 11828 raft_consensus.cc:482] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.195060 11828 raft_consensus.cc:504] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } }
I20240501 20:10:38.195801 11828 leader_election.cc:288] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:38.199213 12243 raft_consensus.cc:504] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:38.200690 12243 leader_election.cc:288] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 2 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:38.201488 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3809cc8ffe6b4a9e99ebbb9d26a65279" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
W20240501 20:10:38.202366 7222 leader_election.cc:334] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.202513 7222 leader_election.cc:302] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97; no voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.203053 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79af181b5f014a40a4c123b47a720dec" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:10:38.203228 4498 raft_consensus.cc:3037] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:38.206002 4498 raft_consensus.cc:2454] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 2.
I20240501 20:10:38.206750 11828 raft_consensus.cc:2726] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:38.208482 4301 leader_election.cc:302] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
W20240501 20:10:38.209671 4301 leader_election.cc:334] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.209954 12243 raft_consensus.cc:2781] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:38.210119 12243 raft_consensus.cc:686] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [term 2 LEADER]: Becoming Leader. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Running, Role: LEADER
W20240501 20:10:38.210979 7090 consensus_peers.cc:462] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.210494 12243 consensus_queue.cc:237] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:38.223729 12205 raft_consensus.cc:482] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.223982 12205 raft_consensus.cc:504] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:38.224736 12205 leader_election.cc:288] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:38.226608 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ecf2962444644cc8c5bdf5caf95c8c8" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:38.228966 12205 raft_consensus.cc:482] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.229204 12205 raft_consensus.cc:504] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:38.229926 11146 raft_consensus.cc:482] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.229993 12205 leader_election.cc:288] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:38.230134 11146 raft_consensus.cc:504] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
W20240501 20:10:38.230502 7355 leader_election.cc:334] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.230672 7355 leader_election.cc:302] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5; no voters: bceeae1c79964acabdae8ff771bf0c97, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.231053 11146 leader_election.cc:288] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
W20240501 20:10:38.233014 4169 leader_election.cc:334] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.234755 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4471aeb3f16f49d9bac12befd4cfd7e7" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:38.235543 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743d7fd2ae014352a5681c2a8ee4447d" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:38.236402 4170 leader_election.cc:302] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dca57823003e4729b769ca12367d7530; no voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.236883 11146 raft_consensus.cc:2726] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:38.237309 12205 raft_consensus.cc:2726] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:10:38.239400 7355 leader_election.cc:334] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.239571 7355 leader_election.cc:302] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5; no voters: dca57823003e4729b769ca12367d7530, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.239820 12205 raft_consensus.cc:2726] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:38.242528 12149 raft_consensus.cc:482] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.242794 12149 raft_consensus.cc:504] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:38.243633 12149 leader_election.cc:288] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:38.245035 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bd0e9f4cd7496785ad71480d7d3d61" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
W20240501 20:10:38.247467 7090 leader_election.cc:334] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.247609 7090 leader_election.cc:302] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db0742ee3860468590eefef6ae6074fd; no voters: 3e974c03d3c546f9ab6c91e762a5426b, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.248267 12149 raft_consensus.cc:2726] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:38.248988 4089 catalog_manager.cc:5484] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a reported cstate change: term changed from 1 to 2, leader changed from ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) to 10b47782558c41ee91f5e164a372049a (127.29.219.66). New cstate: current_term: 2 leader_uuid: "10b47782558c41ee91f5e164a372049a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:38.250342 11146 raft_consensus.cc:482] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.250512 11146 raft_consensus.cc:504] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:38.251411 11146 leader_election.cc:288] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:38.253371 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd21a0faa3bb487cb80f0951b12b3d67" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:38.253794 4367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd21a0faa3bb487cb80f0951b12b3d67" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
W20240501 20:10:38.254334 4169 leader_election.cc:334] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.254806 4169 leader_election.cc:302] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: dca57823003e4729b769ca12367d7530; no voters: 10b47782558c41ee91f5e164a372049a, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.256337 11146 raft_consensus.cc:2726] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:38.264640 11948 raft_consensus.cc:482] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.264816 11948 raft_consensus.cc:504] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:38.265647 11948 leader_election.cc:288] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:38.266677 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d860bb204134f0994462275519feecf" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:38.266949 4233 raft_consensus.cc:2454] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:10:38.267761 4434 leader_election.cc:302] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:38.268321 11948 raft_consensus.cc:2781] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:10:38.268412 11948 raft_consensus.cc:482] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.268489 11948 raft_consensus.cc:3037] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:10:38.269165 4434 leader_election.cc:334] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.271477 11948 raft_consensus.cc:504] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:38.272387 11948 leader_election.cc:288] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:38.275246 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d860bb204134f0994462275519feecf" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:10:38.275465 4233 raft_consensus.cc:3037] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:38.278250 4233 raft_consensus.cc:2454] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 2.
I20240501 20:10:38.279085 4610 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:38.279245 4434 leader_election.cc:302] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530; no voters:
W20240501 20:10:38.280215 4434 leader_election.cc:334] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.287402 11146 raft_consensus.cc:482] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.287629 11146 raft_consensus.cc:504] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:38.289350 11146 leader_election.cc:288] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:38.292315 11999 raft_consensus.cc:482] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.292526 11999 raft_consensus.cc:504] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } }
I20240501 20:10:38.293571 11999 leader_election.cc:288] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:38.293924 11050 raft_consensus.cc:2781] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:38.294124 11050 raft_consensus.cc:686] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [term 2 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:10:38.294487 11050 consensus_queue.cc:237] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
W20240501 20:10:38.297058 4169 leader_election.cc:334] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.297631 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3809cc8ffe6b4a9e99ebbb9d26a65279" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:38.297942 7286 raft_consensus.cc:2454] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:10:38.298462 4563 leader_election.cc:302] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:38.298869 11999 raft_consensus.cc:2781] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:10:38.298949 11999 raft_consensus.cc:482] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting leader election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.299015 11999 raft_consensus.cc:3037] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:38.301811 11999 raft_consensus.cc:504] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } }
I20240501 20:10:38.303823 11999 leader_election.cc:288] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:38.305272 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3809cc8ffe6b4a9e99ebbb9d26a65279" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:38.305491 7284 raft_consensus.cc:3037] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:38.308370 7284 raft_consensus.cc:2454] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 2.
I20240501 20:10:38.308851 4563 leader_election.cc:302] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:38.309219 11999 raft_consensus.cc:2781] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:38.309463 11999 raft_consensus.cc:686] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 LEADER]: Becoming Leader. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Running, Role: LEADER
I20240501 20:10:38.309859 11999 consensus_queue.cc:237] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } }
I20240501 20:10:38.311023 11828 raft_consensus.cc:482] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.311203 11828 raft_consensus.cc:504] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:38.311940 11828 leader_election.cc:288] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
W20240501 20:10:38.313803 7222 leader_election.cc:334] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.316138 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afb8e3154a64be78b8041b137b3681a" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:38.317349 7222 leader_election.cc:302] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97; no voters: 3e974c03d3c546f9ab6c91e762a5426b, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.317904 11828 raft_consensus.cc:2726] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:38.320331 4346 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:38.321905 4089 catalog_manager.cc:5484] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 1 to 2, leader changed from ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 2 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:38.325153 12205 raft_consensus.cc:482] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.325317 12205 raft_consensus.cc:504] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:38.326025 12205 leader_election.cc:288] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
W20240501 20:10:38.327889 7355 leader_election.cc:334] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.328444 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "accaff8513134f96b5502b5c5385d892" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:38.328691 4233 raft_consensus.cc:2454] T accaff8513134f96b5502b5c5385d892 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:38.329358 7355 leader_election.cc:302] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, dca57823003e4729b769ca12367d7530; no voters: ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.329905 12205 raft_consensus.cc:2781] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:10:38.330026 12205 raft_consensus.cc:482] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.330111 12205 raft_consensus.cc:3037] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:38.330901 4478 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 20:10:38.332909 12205 raft_consensus.cc:504] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:38.333640 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4471aeb3f16f49d9bac12befd4cfd7e7" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:38.333870 7419 raft_consensus.cc:2454] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 1.
I20240501 20:10:38.336796 7267 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 20:10:38.337265 7355 leader_election.cc:334] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.337489 11050 raft_consensus.cc:482] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.337659 11050 raft_consensus.cc:504] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:38.338310 11050 leader_election.cc:288] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:38.338604 4170 leader_election.cc:302] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, dca57823003e4729b769ca12367d7530; no voters: ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.339008 11910 raft_consensus.cc:2781] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:10:38.338975 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bd0e9f4cd7496785ad71480d7d3d61" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:38.339094 11910 raft_consensus.cc:482] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.339179 11910 raft_consensus.cc:3037] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:38.339233 7155 raft_consensus.cc:2454] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:10:38.339634 4432 leader_election.cc:302] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:38.341403 11050 raft_consensus.cc:2781] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:10:38.341943 11910 raft_consensus.cc:504] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:38.342078 11050 raft_consensus.cc:482] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.342929 11050 raft_consensus.cc:3037] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:10:38.345407 4434 leader_election.cc:334] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.345705 12205 leader_election.cc:288] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:38.346292 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4471aeb3f16f49d9bac12befd4cfd7e7" candidate_uuid: "dca57823003e4729b769ca12367d7530" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:38.346477 7419 raft_consensus.cc:3037] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:38.347702 11050 raft_consensus.cc:504] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:38.347937 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "accaff8513134f96b5502b5c5385d892" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:10:38.348142 4233 raft_consensus.cc:3037] T accaff8513134f96b5502b5c5385d892 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:10:38.348881 4564 leader_election.cc:334] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.350077 7419 raft_consensus.cc:2454] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dca57823003e4729b769ca12367d7530 in term 2.
I20240501 20:10:38.350989 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99bd0e9f4cd7496785ad71480d7d3d61" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:38.351220 7155 raft_consensus.cc:3037] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:38.351939 4233 raft_consensus.cc:2454] T accaff8513134f96b5502b5c5385d892 P dca57823003e4729b769ca12367d7530 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 2.
I20240501 20:10:38.355139 7155 raft_consensus.cc:2454] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 2.
I20240501 20:10:38.355836 7135 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 20:10:38.356222 4432 leader_election.cc:302] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:38.356662 11948 raft_consensus.cc:2781] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:38.357231 11948 raft_consensus.cc:686] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:10:38.357316 4170 leader_election.cc:302] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:38.357594 11146 raft_consensus.cc:2781] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:38.358093 11146 raft_consensus.cc:686] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530 [term 2 LEADER]: Becoming Leader. State: Replica: dca57823003e4729b769ca12367d7530, State: Running, Role: LEADER
W20240501 20:10:38.358163 4434 leader_election.cc:334] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.358357 11146 consensus_queue.cc:237] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:38.359766 11948 consensus_queue.cc:237] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } }
I20240501 20:10:38.364713 12241 raft_consensus.cc:482] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.364871 12241 raft_consensus.cc:504] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:38.365470 12241 leader_election.cc:288] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:38.365444 7355 leader_election.cc:302] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, dca57823003e4729b769ca12367d7530; no voters: ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.356225 11050 leader_election.cc:288] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 election: Requested vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:38.366204 12205 raft_consensus.cc:2781] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:38.366357 12205 raft_consensus.cc:686] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 LEADER]: Becoming Leader. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Running, Role: LEADER
I20240501 20:10:38.366643 12205 consensus_queue.cc:237] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:38.367832 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afb8e3154a64be78b8041b137b3681a" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:38.367926 11910 leader_election.cc:288] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 2 election: Requested vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:38.368213 7286 raft_consensus.cc:2454] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:10:38.368676 4433 leader_election.cc:302] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, bceeae1c79964acabdae8ff771bf0c97; no voters:
W20240501 20:10:38.370808 4169 leader_election.cc:334] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.374073 12241 raft_consensus.cc:2781] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:10:38.374171 12241 raft_consensus.cc:482] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.374255 12241 raft_consensus.cc:3037] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:38.376232 11828 raft_consensus.cc:482] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.376374 11828 raft_consensus.cc:504] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:38.377045 11828 leader_election.cc:288] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:38.377810 12241 raft_consensus.cc:504] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
W20240501 20:10:38.380501 4434 leader_election.cc:334] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
W20240501 20:10:38.381378 4564 leader_election.cc:334] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.381769 12241 leader_election.cc:288] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 election: Requested vote from peers bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:38.382094 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afb8e3154a64be78b8041b137b3681a" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97"
I20240501 20:10:38.382513 7286 raft_consensus.cc:3037] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:10:38.383955 4434 leader_election.cc:334] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.386876 7286 raft_consensus.cc:2454] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 2.
I20240501 20:10:38.387326 4433 leader_election.cc:302] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, bceeae1c79964acabdae8ff771bf0c97; no voters: ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.387686 7417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ecf2962444644cc8c5bdf5caf95c8c8" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:38.387902 7417 raft_consensus.cc:2454] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:38.388669 7223 leader_election.cc:302] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:38.389345 11828 raft_consensus.cc:2781] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:10:38.389457 11828 raft_consensus.cc:482] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.389551 11828 raft_consensus.cc:3037] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:38.392568 11828 raft_consensus.cc:504] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:38.393222 12241 raft_consensus.cc:2781] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:38.393416 12241 raft_consensus.cc:686] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b [term 2 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:10:38.393815 12241 consensus_queue.cc:237] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
W20240501 20:10:38.402340 7222 leader_election.cc:334] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.403281 11828 leader_election.cc:288] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
W20240501 20:10:38.404706 7222 leader_election.cc:334] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.406318 7417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ecf2962444644cc8c5bdf5caf95c8c8" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:38.406533 7417 raft_consensus.cc:3037] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:38.409382 7417 raft_consensus.cc:2454] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 2.
I20240501 20:10:38.409987 7223 leader_election.cc:302] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, bceeae1c79964acabdae8ff771bf0c97; no voters: ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.410152 12149 raft_consensus.cc:482] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.410424 11828 raft_consensus.cc:2781] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:38.410315 12149 raft_consensus.cc:504] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:38.410624 11828 raft_consensus.cc:686] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97 [term 2 LEADER]: Becoming Leader. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Running, Role: LEADER
I20240501 20:10:38.411010 11828 consensus_queue.cc:237] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:38.411233 12149 leader_election.cc:288] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:38.411649 12159 raft_consensus.cc:482] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.411862 12159 raft_consensus.cc:504] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:38.412819 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd21a0faa3bb487cb80f0951b12b3d67" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a" is_pre_election: true
I20240501 20:10:38.412920 12159 leader_election.cc:288] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:38.413053 4368 raft_consensus.cc:2454] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:38.414213 4232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd21a0faa3bb487cb80f0951b12b3d67" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:38.414314 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd21a0faa3bb487cb80f0951b12b3d67" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:38.414471 4232 raft_consensus.cc:2454] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
I20240501 20:10:38.414496 7153 raft_consensus.cc:2454] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b47782558c41ee91f5e164a372049a in term 1.
W20240501 20:10:38.415781 7090 leader_election.cc:334] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.415895 4301 leader_election.cc:302] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, db0742ee3860468590eefef6ae6074fd, dca57823003e4729b769ca12367d7530; no voters:
W20240501 20:10:38.418135 4301 leader_election.cc:334] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.419793 12159 raft_consensus.cc:2781] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:10:38.419885 12159 raft_consensus.cc:482] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.419953 12159 raft_consensus.cc:3037] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:38.419957 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd21a0faa3bb487cb80f0951b12b3d67" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:38.420166 4233 raft_consensus.cc:2454] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:38.420483 7090 leader_election.cc:302] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 10b47782558c41ee91f5e164a372049a, db0742ee3860468590eefef6ae6074fd, dca57823003e4729b769ca12367d7530; no voters: ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.420892 12149 raft_consensus.cc:2781] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:10:38.420965 12149 raft_consensus.cc:482] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting leader election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.421066 12149 raft_consensus.cc:3037] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:38.422839 12159 raft_consensus.cc:504] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:38.426550 12149 raft_consensus.cc:504] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
W20240501 20:10:38.427057 4301 leader_election.cc:334] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.428053 12439 ts_tablet_manager.cc:902] T aecca0e22b084a8b95c98ef7e7b90b29 P 10b47782558c41ee91f5e164a372049a: Initiating tablet copy from peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:38.428160 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd21a0faa3bb487cb80f0951b12b3d67" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:38.428418 7153 raft_consensus.cc:2379] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 10b47782558c41ee91f5e164a372049a in current term 2: Already voted for candidate db0742ee3860468590eefef6ae6074fd in this term.
W20240501 20:10:38.430389 7090 leader_election.cc:334] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.431121 12149 leader_election.cc:288] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:38.432116 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd21a0faa3bb487cb80f0951b12b3d67" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:10:38.432374 4233 raft_consensus.cc:3037] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:38.432909 4089 catalog_manager.cc:5484] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: term changed from 1 to 2, leader changed from ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) to 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68). New cstate: current_term: 2 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:38.433918 4084 catalog_manager.cc:5484] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: term changed from 1 to 2, leader changed from ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) to bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71). New cstate: current_term: 2 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:38.434363 11999 raft_consensus.cc:482] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.434504 11999 raft_consensus.cc:504] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:38.430449 4299 leader_election.cc:302] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 10b47782558c41ee91f5e164a372049a; no voters: db0742ee3860468590eefef6ae6074fd, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.434484 4088 catalog_manager.cc:5484] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: term changed from 1 to 2, leader changed from ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) to b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72). New cstate: current_term: 2 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:38.434891 12243 raft_consensus.cc:2726] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:10:38.435146 11999 leader_election.cc:288] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
W20240501 20:10:38.436914 4564 leader_election.cc:334] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.438489 12159 leader_election.cc:288] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [CANDIDATE]: Term 2 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:38.438722 12379 raft_consensus.cc:482] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.438863 12379 raft_consensus.cc:504] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:38.439386 12379 leader_election.cc:288] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:38.439741 4232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd21a0faa3bb487cb80f0951b12b3d67" candidate_uuid: "10b47782558c41ee91f5e164a372049a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:10:38.440338 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f63c61d85524d87a4b911c321f486c4" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:38.440577 4498 raft_consensus.cc:2454] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:10:38.440985 4233 raft_consensus.cc:2454] T cd21a0faa3bb487cb80f0951b12b3d67 P dca57823003e4729b769ca12367d7530 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 2.
I20240501 20:10:38.441418 4564 leader_election.cc:302] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters: ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.441416 4368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd21a0faa3bb487cb80f0951b12b3d67" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "10b47782558c41ee91f5e164a372049a"
I20240501 20:10:38.441709 4368 raft_consensus.cc:2379] T cd21a0faa3bb487cb80f0951b12b3d67 P 10b47782558c41ee91f5e164a372049a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate db0742ee3860468590eefef6ae6074fd in current term 2: Already voted for candidate 10b47782558c41ee91f5e164a372049a in this term.
I20240501 20:10:38.441850 11999 raft_consensus.cc:2781] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:10:38.441928 11999 raft_consensus.cc:482] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting leader election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.441998 11999 raft_consensus.cc:3037] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:38.445639 7090 leader_election.cc:302] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: db0742ee3860468590eefef6ae6074fd, dca57823003e4729b769ca12367d7530; no voters: 10b47782558c41ee91f5e164a372049a, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.446269 12149 raft_consensus.cc:2726] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:10:38.450763 12431 raft_consensus.cc:482] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.450942 11050 raft_consensus.cc:482] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.451076 12431 raft_consensus.cc:504] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:38.451861 12431 leader_election.cc:288] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
W20240501 20:10:38.454131 4434 leader_election.cc:334] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.454362 4232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f13fda83114ec5b9dcd0fa90ddef7e" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:38.454597 4232 raft_consensus.cc:2454] T 12f13fda83114ec5b9dcd0fa90ddef7e P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:38.455333 7355 leader_election.cc:302] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, dca57823003e4729b769ca12367d7530; no voters:
W20240501 20:10:38.455580 7355 leader_election.cc:334] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.456001 12205 raft_consensus.cc:2781] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:10:38.456112 12205 raft_consensus.cc:482] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.456194 12205 raft_consensus.cc:3037] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:38.456667 4089 catalog_manager.cc:5484] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 1 to 2, leader changed from ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 2 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:38.459085 12205 raft_consensus.cc:504] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:38.459681 12205 leader_election.cc:288] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 election: Requested vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:38.462832 4232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f13fda83114ec5b9dcd0fa90ddef7e" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:10:38.463035 4232 raft_consensus.cc:3037] T 12f13fda83114ec5b9dcd0fa90ddef7e P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:10:38.463733 7355 leader_election.cc:334] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.465840 4232 raft_consensus.cc:2454] T 12f13fda83114ec5b9dcd0fa90ddef7e P dca57823003e4729b769ca12367d7530 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 2.
I20240501 20:10:38.466503 12119 raft_consensus.cc:482] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.466671 12119 raft_consensus.cc:504] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:38.467463 12119 leader_election.cc:288] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:38.471704 7355 leader_election.cc:302] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, dca57823003e4729b769ca12367d7530; no voters: ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.472008 12205 raft_consensus.cc:2781] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:38.472174 12205 raft_consensus.cc:686] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [term 2 LEADER]: Becoming Leader. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Running, Role: LEADER
I20240501 20:10:38.451064 11050 raft_consensus.cc:504] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:38.472458 12205 consensus_queue.cc:237] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:38.473030 11050 leader_election.cc:288] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:38.450163 11999 raft_consensus.cc:504] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:38.475488 11999 leader_election.cc:288] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
W20240501 20:10:38.479692 4434 leader_election.cc:334] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.483533 12439 tablet_copy_client.cc:308] T aecca0e22b084a8b95c98ef7e7b90b29 P 10b47782558c41ee91f5e164a372049a: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.71:39447
I20240501 20:10:38.484283 7308 tablet_copy_service.cc:140] P bceeae1c79964acabdae8ff771bf0c97: Received BeginTabletCopySession request for tablet aecca0e22b084a8b95c98ef7e7b90b29 from peer 10b47782558c41ee91f5e164a372049a ({username='slave'} at 127.29.219.66:32801)
I20240501 20:10:38.484477 7308 tablet_copy_service.cc:161] P bceeae1c79964acabdae8ff771bf0c97: Beginning new tablet copy session on tablet aecca0e22b084a8b95c98ef7e7b90b29 from peer 10b47782558c41ee91f5e164a372049a at {username='slave'} at 127.29.219.66:32801: session id = 10b47782558c41ee91f5e164a372049a-aecca0e22b084a8b95c98ef7e7b90b29
I20240501 20:10:38.491243 4215 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 20:10:38.491578 7308 tablet_copy_source_session.cc:205] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:38.491801 4089 catalog_manager.cc:5484] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530 reported cstate change: term changed from 1 to 2, leader changed from ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) to dca57823003e4729b769ca12367d7530 (127.29.219.65). New cstate: current_term: 2 leader_uuid: "dca57823003e4729b769ca12367d7530" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:38.493724 4084 catalog_manager.cc:5484] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: term changed from 1 to 2, leader changed from ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) to b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72). New cstate: current_term: 2 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:38.499204 12439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aecca0e22b084a8b95c98ef7e7b90b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:38.501459 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48528ec242974de4ac20eb0161f0cb05" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:38.501714 7155 raft_consensus.cc:2454] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:10:38.502669 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b2082919b548bfb56b939373caf770" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:38.502882 7155 raft_consensus.cc:2454] T 56b2082919b548bfb56b939373caf770 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 1.
I20240501 20:10:38.506273 4495 raft_consensus.cc:1261] T f18af954cefa48d4882313fe18abbd27 P 3e974c03d3c546f9ab6c91e762a5426b [term 3 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240501 20:10:38.512648 4495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f63c61d85524d87a4b911c321f486c4" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b"
I20240501 20:10:38.512881 4495 raft_consensus.cc:3037] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:38.513545 4084 catalog_manager.cc:5484] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 1 to 2, leader changed from ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 2 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:38.516335 4495 raft_consensus.cc:2454] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 2.
I20240501 20:10:38.516808 4432 leader_election.cc:302] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, db0742ee3860468590eefef6ae6074fd; no voters: ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.518114 11948 raft_consensus.cc:2781] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:10:38.518270 11948 raft_consensus.cc:482] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.518543 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743d7fd2ae014352a5681c2a8ee4447d" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530" is_pre_election: true
I20240501 20:10:38.518811 11948 raft_consensus.cc:3037] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:38.519681 4233 raft_consensus.cc:2454] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:10:38.522290 12439 tablet_copy_client.cc:783] T aecca0e22b084a8b95c98ef7e7b90b29 P 10b47782558c41ee91f5e164a372049a: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:38.522544 12439 tablet_copy_client.cc:647] T aecca0e22b084a8b95c98ef7e7b90b29 P 10b47782558c41ee91f5e164a372049a: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:38.525971 4432 leader_election.cc:302] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, db0742ee3860468590eefef6ae6074fd; no voters: ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.529372 12435 raft_consensus.cc:2781] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:10:38.529527 12435 raft_consensus.cc:482] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting leader election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.529695 12435 raft_consensus.cc:3037] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:38.531386 12439 tablet_copy_client.cc:523] T aecca0e22b084a8b95c98ef7e7b90b29 P 10b47782558c41ee91f5e164a372049a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:38.536046 7400 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 20:10:38.539942 4564 leader_election.cc:302] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:38.541059 12435 raft_consensus.cc:504] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:38.542837 11999 raft_consensus.cc:2781] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:10:38.542981 11999 raft_consensus.cc:482] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting leader election (detected failure of leader ef0a4ada548446b3a531cb39a3c4e38d)
I20240501 20:10:38.543130 11999 raft_consensus.cc:3037] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:38.544247 12439 tablet_bootstrap.cc:492] T aecca0e22b084a8b95c98ef7e7b90b29 P 10b47782558c41ee91f5e164a372049a: Bootstrap starting.
I20240501 20:10:38.545912 11999 raft_consensus.cc:504] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:38.546523 11999 leader_election.cc:288] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:38.547658 12119 consensus_queue.cc:1035] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:10:38.548262 4564 leader_election.cc:302] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters:
I20240501 20:10:38.548955 12119 raft_consensus.cc:2781] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:38.549113 12119 raft_consensus.cc:686] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 LEADER]: Becoming Leader. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Running, Role: LEADER
I20240501 20:10:38.549355 12119 consensus_queue.cc:237] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:38.556483 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "743d7fd2ae014352a5681c2a8ee4447d" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dca57823003e4729b769ca12367d7530"
I20240501 20:10:38.556725 4233 raft_consensus.cc:3037] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:38.559854 4233 raft_consensus.cc:2454] T 743d7fd2ae014352a5681c2a8ee4447d P dca57823003e4729b769ca12367d7530 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 2.
W20240501 20:10:38.562232 4434 leader_election.cc:334] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.563024 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56b2082919b548bfb56b939373caf770" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:38.563230 7155 raft_consensus.cc:3037] T 56b2082919b548bfb56b939373caf770 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:38.565981 7155 raft_consensus.cc:2454] T 56b2082919b548bfb56b939373caf770 P db0742ee3860468590eefef6ae6074fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 2.
I20240501 20:10:38.574512 4432 leader_election.cc:302] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, db0742ee3860468590eefef6ae6074fd; no voters: ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.574973 12432 raft_consensus.cc:2781] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:38.575502 12432 raft_consensus.cc:686] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:10:38.575923 12432 consensus_queue.cc:237] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:38.548697 11948 raft_consensus.cc:504] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
W20240501 20:10:38.577606 4564 leader_election.cc:334] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.577476 12435 leader_election.cc:288] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 election: Requested vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
W20240501 20:10:38.577857 4564 leader_election.cc:334] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.578433 4084 catalog_manager.cc:5484] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: term changed from 1 to 2, leader changed from ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) to 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68). New cstate: current_term: 2 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:38.579123 4564 leader_election.cc:302] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530; no voters:
I20240501 20:10:38.579578 12119 raft_consensus.cc:2781] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Leader election won for term 2
W20240501 20:10:38.579821 4434 leader_election.cc:334] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.580190 11948 leader_election.cc:288] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 election: Requested vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:38.580715 7155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48528ec242974de4ac20eb0161f0cb05" candidate_uuid: "3e974c03d3c546f9ab6c91e762a5426b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:38.580940 7155 raft_consensus.cc:3037] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:38.583998 7155 raft_consensus.cc:2454] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e974c03d3c546f9ab6c91e762a5426b in term 2.
I20240501 20:10:38.593883 4084 catalog_manager.cc:5484] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 1 to 2, leader changed from ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 2 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:10:38.598456 4301 consensus_peers.cc:485] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:38.607860 12439 tablet_bootstrap.cc:492] T aecca0e22b084a8b95c98ef7e7b90b29 P 10b47782558c41ee91f5e164a372049a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:38.608575 12439 tablet_bootstrap.cc:492] T aecca0e22b084a8b95c98ef7e7b90b29 P 10b47782558c41ee91f5e164a372049a: Bootstrap complete.
I20240501 20:10:38.609162 12439 ts_tablet_manager.cc:1367] T aecca0e22b084a8b95c98ef7e7b90b29 P 10b47782558c41ee91f5e164a372049a: Time spent bootstrapping tablet: real 0.065s user 0.012s sys 0.000s
I20240501 20:10:38.611596 4432 leader_election.cc:302] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, db0742ee3860468590eefef6ae6074fd; no voters: ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:38.611951 12432 raft_consensus.cc:2781] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:38.612164 12432 raft_consensus.cc:686] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 LEADER]: Becoming Leader. State: Replica: 3e974c03d3c546f9ab6c91e762a5426b, State: Running, Role: LEADER
I20240501 20:10:38.611920 12439 raft_consensus.cc:348] T aecca0e22b084a8b95c98ef7e7b90b29 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:38.612479 12439 raft_consensus.cc:729] T aecca0e22b084a8b95c98ef7e7b90b29 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 10b47782558c41ee91f5e164a372049a, State: Initialized, Role: LEARNER
I20240501 20:10:38.612660 12432 consensus_queue.cc:237] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:38.613123 12439 consensus_queue.cc:260] T aecca0e22b084a8b95c98ef7e7b90b29 P 10b47782558c41ee91f5e164a372049a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: NON_VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: true } }
I20240501 20:10:38.613266 12119 raft_consensus.cc:686] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 LEADER]: Becoming Leader. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Running, Role: LEADER
I20240501 20:10:38.614271 12119 consensus_queue.cc:237] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
W20240501 20:10:38.619375 4564 leader_election.cc:334] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.621171 12439 ts_tablet_manager.cc:1398] T aecca0e22b084a8b95c98ef7e7b90b29 P 10b47782558c41ee91f5e164a372049a: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240501 20:10:38.627094 7308 tablet_copy_service.cc:342] P bceeae1c79964acabdae8ff771bf0c97: Request end of tablet copy session 10b47782558c41ee91f5e164a372049a-aecca0e22b084a8b95c98ef7e7b90b29 received from {username='slave'} at 127.29.219.66:32801
I20240501 20:10:38.627229 7308 tablet_copy_service.cc:433] P bceeae1c79964acabdae8ff771bf0c97: ending tablet copy session 10b47782558c41ee91f5e164a372049a-aecca0e22b084a8b95c98ef7e7b90b29 on tablet aecca0e22b084a8b95c98ef7e7b90b29 with peer 10b47782558c41ee91f5e164a372049a
I20240501 20:10:38.628309 4089 catalog_manager.cc:5484] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e reported cstate change: term changed from 1 to 2, leader changed from ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) to 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68). New cstate: current_term: 2 leader_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 20:10:38.636202 4563 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.71:39447, user_credentials={real_user=slave}} blocked reactor thread for 50530.4us
W20240501 20:10:38.650123 4564 consensus_peers.cc:485] T f18af954cefa48d4882313fe18abbd27 P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:38.653307 4084 catalog_manager.cc:5484] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: term changed from 1 to 2, leader changed from ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) to 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67). New cstate: current_term: 2 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:10:38.681314 7090 consensus_peers.cc:462] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.717561 4233 raft_consensus.cc:1261] T 1d860bb204134f0994462275519feecf P dca57823003e4729b769ca12367d7530 [term 2 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:10:38.718230 12432 consensus_queue.cc:1035] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 20:10:38.736346 4434 consensus_peers.cc:485] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:38.750566 4495 raft_consensus.cc:1261] T 79af181b5f014a40a4c123b47a720dec P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Refusing update from remote peer 10b47782558c41ee91f5e164a372049a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 20:10:38.751240 12159 consensus_queue.cc:1035] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 20:10:38.758103 7355 consensus_peers.cc:485] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:38.767392 4233 raft_consensus.cc:1261] T accaff8513134f96b5502b5c5385d892 P dca57823003e4729b769ca12367d7530 [term 2 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:10:38.768136 12205 consensus_queue.cc:1035] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:38.774881 4564 consensus_peers.cc:485] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:38.793208 4302 consensus_peers.cc:485] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a -> Peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Couldn't send request to peer 1d34e18a1d394d3f91c9f50d1c7bea4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79af181b5f014a40a4c123b47a720dec. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:38.798655 7417 raft_consensus.cc:3037] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 LEARNER]: Advancing to term 2
I20240501 20:10:38.802867 7417 raft_consensus.cc:1261] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 LEARNER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:10:38.804458 12432 consensus_queue.cc:1035] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: NON_VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:38.817323 12435 raft_consensus.cc:1050] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b: attempting to promote NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 to VOTER
W20240501 20:10:38.817509 12435 raft_consensus.cc:1054] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b: Unable to promote non-voter b9957d4086bc4a76b18ed9265a5739d5: Illegal state: Leader has not yet committed an operation in its own term
I20240501 20:10:38.818390 7284 raft_consensus.cc:1261] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:10:38.818969 12453 consensus_queue.cc:1035] T 3809cc8ffe6b4a9e99ebbb9d26a65279 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:38.834103 7286 raft_consensus.cc:1261] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:10:38.835678 12435 consensus_queue.cc:1035] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 20:10:38.842173 4434 consensus_peers.cc:485] T 7afb8e3154a64be78b8041b137b3681a P 3e974c03d3c546f9ab6c91e762a5426b -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:38.847342 7153 raft_consensus.cc:1261] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd [term 2 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:10:38.847965 12435 consensus_queue.cc:1035] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 20:10:38.853682 4434 consensus_peers.cc:485] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:38.856752 7419 raft_consensus.cc:1261] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Refusing update from remote peer dca57823003e4729b769ca12367d7530: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:10:38.857388 11146 consensus_queue.cc:1035] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 20:10:38.866380 4169 consensus_peers.cc:485] T 4471aeb3f16f49d9bac12befd4cfd7e7 P dca57823003e4729b769ca12367d7530 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:38.869652 12435 raft_consensus.cc:1050] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b: attempting to promote NON_VOTER b9957d4086bc4a76b18ed9265a5739d5 to VOTER
I20240501 20:10:38.870249 12435 consensus_queue.cc:237] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:38.872810 7419 raft_consensus.cc:1261] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 LEARNER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:10:38.872901 7153 raft_consensus.cc:1261] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd [term 2 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:10:38.873368 12431 consensus_queue.cc:1035] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:38.873956 12431 consensus_queue.cc:1035] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:10:38.874722 4434 consensus_peers.cc:485] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:38.877918 12431 raft_consensus.cc:2932] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:38.882335 7155 raft_consensus.cc:2932] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:38.882313 7417 raft_consensus.cc:2932] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:38.884879 4084 catalog_manager.cc:5484] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 2 to 4, b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:38.890115 4495 consensus_queue.cc:237] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:38.892925 7155 raft_consensus.cc:1261] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd [term 2 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:10:38.893527 11050 consensus_queue.cc:1035] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:38.900624 7419 raft_consensus.cc:1261] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240501 20:10:38.902953 7355 consensus_peers.cc:485] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:38.903537 12431 consensus_queue.cc:1035] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:38.909389 12435 raft_consensus.cc:2932] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:38.911098 4233 raft_consensus.cc:1261] T 12f13fda83114ec5b9dcd0fa90ddef7e P dca57823003e4729b769ca12367d7530 [term 2 FOLLOWER]: Refusing update from remote peer b9957d4086bc4a76b18ed9265a5739d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:10:38.912040 7419 raft_consensus.cc:2932] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:38.913906 12205 consensus_queue.cc:1035] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:38.917461 4078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 99bd0e9f4cd7496785ad71480d7d3d61 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:38.926249 4368 raft_consensus.cc:1203] T aecca0e22b084a8b95c98ef7e7b90b29 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 20:10:38.933835 7155 raft_consensus.cc:2932] T 99bd0e9f4cd7496785ad71480d7d3d61 P db0742ee3860468590eefef6ae6074fd [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } }
I20240501 20:10:38.941198 4088 catalog_manager.cc:5484] T 99bd0e9f4cd7496785ad71480d7d3d61 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 4 to 5, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New cstate: current_term: 2 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:10:38.949110 4078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.954515 7155 raft_consensus.cc:1261] T 56b2082919b548bfb56b939373caf770 P db0742ee3860468590eefef6ae6074fd [term 2 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 20:10:38.955230 12435 consensus_queue.cc:1035] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:38.962836 4627 raft_consensus.cc:3037] T 56b2082919b548bfb56b939373caf770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 LEARNER]: Advancing to term 2
I20240501 20:10:38.965749 4627 raft_consensus.cc:1261] T 56b2082919b548bfb56b939373caf770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 LEARNER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20240501 20:10:38.969352 4434 consensus_peers.cc:485] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:38.970050 12432 consensus_queue.cc:1035] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:38.978933 11050 raft_consensus.cc:1050] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b: attempting to promote NON_VOTER 1d34e18a1d394d3f91c9f50d1c7bea4e to VOTER
I20240501 20:10:38.979727 11050 consensus_queue.cc:237] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } }
I20240501 20:10:38.982343 7152 raft_consensus.cc:1261] T 56b2082919b548bfb56b939373caf770 P db0742ee3860468590eefef6ae6074fd [term 2 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:10:38.982411 4627 raft_consensus.cc:1261] T 56b2082919b548bfb56b939373caf770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 LEARNER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20240501 20:10:38.982607 4434 consensus_peers.cc:485] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:38.982939 12435 consensus_queue.cc:1035] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 20:10:38.984133 12431 consensus_queue.cc:1035] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 20:10:38.989380 4078 catalog_manager.cc:4573] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 99bd0e9f4cd7496785ad71480d7d3d61: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:38.991355 7419 raft_consensus.cc:1261] T 7ecf2962444644cc8c5bdf5caf95c8c8 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 20:10:38.992111 12437 consensus_queue.cc:1035] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 20:10:38.998930 12431 raft_consensus.cc:2932] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:39.002651 4631 raft_consensus.cc:2932] T 56b2082919b548bfb56b939373caf770 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
I20240501 20:10:39.009380 7152 raft_consensus.cc:2932] T 56b2082919b548bfb56b939373caf770 P db0742ee3860468590eefef6ae6074fd [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } } }
W20240501 20:10:39.011076 4434 consensus_peers.cc:485] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:39.013118 4084 catalog_manager.cc:5484] T 56b2082919b548bfb56b939373caf770 P 3e974c03d3c546f9ab6c91e762a5426b reported cstate change: config changed from index 2 to 4, 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "3e974c03d3c546f9ab6c91e762a5426b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:10:39.022352 7222 consensus_peers.cc:485] T 7ecf2962444644cc8c5bdf5caf95c8c8 P bceeae1c79964acabdae8ff771bf0c97 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:39.030320 4495 raft_consensus.cc:1261] T 3f63c61d85524d87a4b911c321f486c4 P 3e974c03d3c546f9ab6c91e762a5426b [term 2 FOLLOWER]: Refusing update from remote peer 1d34e18a1d394d3f91c9f50d1c7bea4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:10:39.030894 11999 consensus_queue.cc:1035] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:10:39.037293 7152 raft_consensus.cc:1261] T 48528ec242974de4ac20eb0161f0cb05 P db0742ee3860468590eefef6ae6074fd [term 2 FOLLOWER]: Refusing update from remote peer 3e974c03d3c546f9ab6c91e762a5426b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:10:39.037809 12431 consensus_queue.cc:1035] T 48528ec242974de4ac20eb0161f0cb05 P 3e974c03d3c546f9ab6c91e762a5426b [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 20:10:39.078140 4564 consensus_peers.cc:485] T 3f63c61d85524d87a4b911c321f486c4 P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:39.081593 7090 consensus_peers.cc:462] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
W20240501 20:10:39.097926 4564 consensus_peers.cc:485] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1203: Failure
Value of: s.ok()
Actual: false
Expected: true
Runtime error: /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 2
Maximum Replica Count | 39
Average Replica Count | 20.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 2
Maximum | 3
Average | 2.944444
stderr: W20240501 20:09:59.784220 7475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:09:59.797580 7475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:09:59.834311 7475 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20240501 20:10:00.318238 7562 meta_cache.cc:1015] Not found: LookupRpc { table: 'rebalance_test_table_2', partition-key: (<start>), attempt: 1 } failed: table rebalance_test_table_2 was deleted: Table deleted at 2024-05-01 20:10:00 UTC
W20240501 20:10:00.895332 7535 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.29.219.65:42941, user_credentials={real_user=slave}} blocked reactor thread for 56120.9us
W20240501 20:10:01.410425 7687 meta_cache.cc:1015] Not found: LookupRpc { table: 'rebalance_test_table_6', partition-key: (<start>), attempt: 1 } failed: table rebalance_test_table_6 was deleted: Table deleted at 2024-05-01 20:10:01 UTC
I20240501 20:10:02.103541 7475 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 20:10:03.105602 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ece9130016fb4578a855bb117efb9bca: could not find any suitable replica to move from server dca57823003e4729b769ca12367d7530 to server b9957d4086bc4a76b18ed9265a5739d5
W20240501 20:10:03.105890 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 061e2b9b5d53488cbfdc25cfa8acdd76: could not find any suitable replica to move from server bceeae1c79964acabdae8ff771bf0c97 to server 3e974c03d3c546f9ab6c91e762a5426b
W20240501 20:10:03.106003 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b02a4c6c0f54681a87f5ca2381de01d: could not find any suitable replica to move from server 10b47782558c41ee91f5e164a372049a to server bceeae1c79964acabdae8ff771bf0c97
W20240501 20:10:03.106181 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ece9130016fb4578a855bb117efb9bca: could not find any suitable replica to move from server 1d34e18a1d394d3f91c9f50d1c7bea4e to server db0742ee3860468590eefef6ae6074fd
W20240501 20:10:03.106240 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 40f3268f46a14344bdf5de27989a6f1a: could not find any suitable replica to move from server db0742ee3860468590eefef6ae6074fd to server 3e974c03d3c546f9ab6c91e762a5426b
W20240501 20:10:03.106305 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 09dd73eba68c4f8499a786750d39c1cf: could not find any suitable replica to move from server 3e974c03d3c546f9ab6c91e762a5426b to server b9957d4086bc4a76b18ed9265a5739d5
W20240501 20:10:03.106413 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b5dc7891d8a1410bb1280ac3ce2fd364: could not find any suitable replica to move from server 10b47782558c41ee91f5e164a372049a to server ef0a4ada548446b3a531cb39a3c4e38d
W20240501 20:10:03.106480 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1e137c7380c64655a3b011211c536d72: could not find any suitable replica to move from server 1d34e18a1d394d3f91c9f50d1c7bea4e to server b9957d4086bc4a76b18ed9265a5739d5
W20240501 20:10:03.106549 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 62c829b3c0b3400bb175722fa826b0f4: could not find any suitable replica to move from server dca57823003e4729b769ca12367d7530 to server bceeae1c79964acabdae8ff771bf0c97
W20240501 20:10:03.106853 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c3c3c4ff4c8444a38dae5dc10a57d879: could not find any suitable replica to move from server 3e974c03d3c546f9ab6c91e762a5426b to server ef0a4ada548446b3a531cb39a3c4e38d
W20240501 20:10:03.106961 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 59dd8401cbd048cea30daf86baac8cfa: could not find any suitable replica to move from server dca57823003e4729b769ca12367d7530 to server db0742ee3860468590eefef6ae6074fd
W20240501 20:10:03.107110 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e21b0047cbf842c6833a78db81a34171: could not find any suitable replica to move from server 1d34e18a1d394d3f91c9f50d1c7bea4e to server b9957d4086bc4a76b18ed9265a5739d5
W20240501 20:10:03.107172 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ece9130016fb4578a855bb117efb9bca: could not find any suitable replica to move from server 10b47782558c41ee91f5e164a372049a to server bceeae1c79964acabdae8ff771bf0c97
W20240501 20:10:03.107290 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eef6790ef9874038b63d81f758f188d7: could not find any suitable replica to move from server 1d34e18a1d394d3f91c9f50d1c7bea4e to server b9957d4086bc4a76b18ed9265a5739d5
W20240501 20:10:03.107352 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b5dc7891d8a1410bb1280ac3ce2fd364: could not find any suitable replica to move from server dca57823003e4729b769ca12367d7530 to server b9957d4086bc4a76b18ed9265a5739d5
W20240501 20:10:03.107465 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eccfbde58fa5455b9f89a20ad3a37f39: could not find any suitable replica to move from server dca57823003e4729b769ca12367d7530 to server b9957d4086bc4a76b18ed9265a5739d5
W20240501 20:10:03.107539 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2478c1c300d14f2a9826930dc16a5dec: could not find any suitable replica to move from server 3e974c03d3c546f9ab6c91e762a5426b to server db0742ee3860468590eefef6ae6074fd
W20240501 20:10:03.107621 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e07c82123bdc4376bae10c48e6e6543a: could not find any suitable replica to move from server 10b47782558c41ee91f5e164a372049a to server ef0a4ada548446b3a531cb39a3c4e38d
W20240501 20:10:03.107681 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8de1e355c457486da8bba4c628d52c17: could not find any suitable replica to move from server dca57823003e4729b769ca12367d7530 to server ef0a4ada548446b3a531cb39a3c4e38d
W20240501 20:10:03.107800 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5fb9ba92237649f89c99e4487cd45f29: could not find any suitable replica to move from server dca57823003e4729b769ca12367d7530 to server db0742ee3860468590eefef6ae6074fd
W20240501 20:10:03.107892 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 09dd73eba68c4f8499a786750d39c1cf: could not find any suitable replica to move from server 1d34e18a1d394d3f91c9f50d1c7bea4e to server bceeae1c79964acabdae8ff771bf0c97
W20240501 20:10:03.107990 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e21b0047cbf842c6833a78db81a34171: could not find any suitable replica to move from server 3e974c03d3c546f9ab6c91e762a5426b to server db0742ee3860468590eefef6ae6074fd
W20240501 20:10:03.108078 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eef6790ef9874038b63d81f758f188d7: could not find any suitable replica to move from server 10b47782558c41ee91f5e164a372049a to server bceeae1c79964acabdae8ff771bf0c97
W20240501 20:10:03.108229 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1e137c7380c64655a3b011211c536d72: could not find any suitable replica to move from server 10b47782558c41ee91f5e164a372049a to server b9957d4086bc4a76b18ed9265a5739d5
I20240501 20:10:03.145676 7475 rebalancer_tool.cc:1074] tablet c31148e9e4dd42c3a11787d9384c79c2: '3e974c03d3c546f9ab6c91e762a5426b' -> 'b9957d4086bc4a76b18ed9265a5739d5' move scheduled
I20240501 20:10:03.225939 7475 rebalancer_tool.cc:1074] tablet 46df0faa681b469abe9e383b181a6053: '10b47782558c41ee91f5e164a372049a' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move scheduled
I20240501 20:10:03.269548 7475 rebalancer_tool.cc:1074] tablet 66c533992f8b4c7a937e79f397636bcc: 'dca57823003e4729b769ca12367d7530' -> 'db0742ee3860468590eefef6ae6074fd' move scheduled
I20240501 20:10:03.355700 7475 rebalancer_tool.cc:1074] tablet dfaa3794205d482c90aa4f860fa99cf3: '1d34e18a1d394d3f91c9f50d1c7bea4e' -> 'bceeae1c79964acabdae8ff771bf0c97' move scheduled
I20240501 20:10:03.394126 7475 rebalancer_tool.cc:1074] tablet 34cb2f3c90cc4158a61f426c0e5cb86b: '3e974c03d3c546f9ab6c91e762a5426b' -> 'b9957d4086bc4a76b18ed9265a5739d5' move scheduled
I20240501 20:10:03.496317 7475 rebalancer_tool.cc:1074] tablet 9d634c1feb7449f686cf3af2a07181e0: '10b47782558c41ee91f5e164a372049a' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move scheduled
I20240501 20:10:03.746913 7475 rebalancer_tool.cc:1084] tablet a215d2ce50c347efb5b536a27d037094: 'dca57823003e4729b769ca12367d7530' -> 'db0742ee3860468590eefef6ae6074fd' move ignored: Not found: tablet not found:: a215d2ce50c347efb5b536a27d037094
I20240501 20:10:03.747190 7475 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 20:10:05.536916 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b5dc7891d8a1410bb1280ac3ce2fd364: could not find any suitable replica to move from server dca57823003e4729b769ca12367d7530 to server ef0a4ada548446b3a531cb39a3c4e38d
W20240501 20:10:05.537222 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b5dc7891d8a1410bb1280ac3ce2fd364: could not find any suitable replica to move from server 1d34e18a1d394d3f91c9f50d1c7bea4e to server db0742ee3860468590eefef6ae6074fd
W20240501 20:10:05.537492 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1e137c7380c64655a3b011211c536d72: could not find any suitable replica to move from server 10b47782558c41ee91f5e164a372049a to server bceeae1c79964acabdae8ff771bf0c97
W20240501 20:10:05.537632 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e21b0047cbf842c6833a78db81a34171: could not find any suitable replica to move from server 3e974c03d3c546f9ab6c91e762a5426b to server bceeae1c79964acabdae8ff771bf0c97
W20240501 20:10:05.537961 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b5dc7891d8a1410bb1280ac3ce2fd364: could not find any suitable replica to move from server 10b47782558c41ee91f5e164a372049a to server b9957d4086bc4a76b18ed9265a5739d5
W20240501 20:10:05.538102 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eef6790ef9874038b63d81f758f188d7: could not find any suitable replica to move from server 1d34e18a1d394d3f91c9f50d1c7bea4e to server db0742ee3860468590eefef6ae6074fd
W20240501 20:10:05.538273 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2478c1c300d14f2a9826930dc16a5dec: could not find any suitable replica to move from server 3e974c03d3c546f9ab6c91e762a5426b to server ef0a4ada548446b3a531cb39a3c4e38d
W20240501 20:10:05.538509 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eccfbde58fa5455b9f89a20ad3a37f39: could not find any suitable replica to move from server 1d34e18a1d394d3f91c9f50d1c7bea4e to server b9957d4086bc4a76b18ed9265a5739d5
W20240501 20:10:05.538606 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 09dd73eba68c4f8499a786750d39c1cf: could not find any suitable replica to move from server 10b47782558c41ee91f5e164a372049a to server bceeae1c79964acabdae8ff771bf0c97
W20240501 20:10:05.538735 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b5dc7891d8a1410bb1280ac3ce2fd364: could not find any suitable replica to move from server dca57823003e4729b769ca12367d7530 to server bceeae1c79964acabdae8ff771bf0c97
W20240501 20:10:05.538911 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 40f3268f46a14344bdf5de27989a6f1a: could not find any suitable replica to move from server db0742ee3860468590eefef6ae6074fd to server 3e974c03d3c546f9ab6c91e762a5426b
W20240501 20:10:05.539036 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3b02a4c6c0f54681a87f5ca2381de01d: could not find any suitable replica to move from server 3e974c03d3c546f9ab6c91e762a5426b to server db0742ee3860468590eefef6ae6074fd
W20240501 20:10:05.539129 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8de1e355c457486da8bba4c628d52c17: could not find any suitable replica to move from server 3e974c03d3c546f9ab6c91e762a5426b to server db0742ee3860468590eefef6ae6074fd
W20240501 20:10:05.539217 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 59dd8401cbd048cea30daf86baac8cfa: could not find any suitable replica to move from server 10b47782558c41ee91f5e164a372049a to server ef0a4ada548446b3a531cb39a3c4e38d
W20240501 20:10:05.539311 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5fb9ba92237649f89c99e4487cd45f29: could not find any suitable replica to move from server 3e974c03d3c546f9ab6c91e762a5426b to server db0742ee3860468590eefef6ae6074fd
W20240501 20:10:05.539417 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2478c1c300d14f2a9826930dc16a5dec: could not find any suitable replica to move from server dca57823003e4729b769ca12367d7530 to server bceeae1c79964acabdae8ff771bf0c97
W20240501 20:10:05.539494 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1e137c7380c64655a3b011211c536d72: could not find any suitable replica to move from server 1d34e18a1d394d3f91c9f50d1c7bea4e to server db0742ee3860468590eefef6ae6074fd
W20240501 20:10:05.539582 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e21b0047cbf842c6833a78db81a34171: could not find any suitable replica to move from server 10b47782558c41ee91f5e164a372049a to server ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:05.786223 7475 rebalancer_tool.cc:1074] tablet 9d165c29dabe4ae3be11ed84702c3cb2: '1d34e18a1d394d3f91c9f50d1c7bea4e' -> 'db0742ee3860468590eefef6ae6074fd' move scheduled
I20240501 20:10:05.966827 7475 rebalancer_tool.cc:1074] tablet f40b062e103d4e54a913fa0f4111f623: 'dca57823003e4729b769ca12367d7530' -> 'bceeae1c79964acabdae8ff771bf0c97' move scheduled
I20240501 20:10:06.023264 7475 rebalancer_tool.cc:1074] tablet 644d4d293e8c462297c6419965bd2c89: '3e974c03d3c546f9ab6c91e762a5426b' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move scheduled
I20240501 20:10:06.055339 7475 rebalancer_tool.cc:1074] tablet ef7c0a48f11945d3953f3461d06bc184: '10b47782558c41ee91f5e164a372049a' -> 'b9957d4086bc4a76b18ed9265a5739d5' move scheduled
I20240501 20:10:06.120566 7475 rebalancer_tool.cc:1074] tablet c66514deec5744da832bc030a582c46a: '1d34e18a1d394d3f91c9f50d1c7bea4e' -> 'bceeae1c79964acabdae8ff771bf0c97' move scheduled
I20240501 20:10:06.192617 7475 rebalancer_tool.cc:1074] tablet 7725fa9f98e9439797a532ed6f2f7dba: '3e974c03d3c546f9ab6c91e762a5426b' -> 'db0742ee3860468590eefef6ae6074fd' move scheduled
I20240501 20:10:06.223330 7475 rebalancer_tool.cc:1074] tablet fcff4be357084bf0903cfb1c0f62a866: 'dca57823003e4729b769ca12367d7530' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move scheduled
I20240501 20:10:06.260674 7475 rebalancer_tool.cc:1074] tablet d690199665174c7d9bdbd54d2af4ac70: '10b47782558c41ee91f5e164a372049a' -> 'b9957d4086bc4a76b18ed9265a5739d5' move scheduled
I20240501 20:10:06.388203 7475 rebalancer_tool.cc:1074] tablet bb85d7825c564339bd554deeded45cec: 'bceeae1c79964acabdae8ff771bf0c97' -> '1d34e18a1d394d3f91c9f50d1c7bea4e' move scheduled
I20240501 20:10:06.574448 7475 rebalancer_tool.cc:1074] tablet 41d6e6dce35e4d0da594f147a7f9bfc4: '3e974c03d3c546f9ab6c91e762a5426b' -> 'db0742ee3860468590eefef6ae6074fd' move scheduled
I20240501 20:10:07.155861 7475 rebalancer_tool.cc:1074] tablet 1c6c89fa7e6f485e889a5197da4bd135: 'dca57823003e4729b769ca12367d7530' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move scheduled
I20240501 20:10:07.557291 7475 rebalancer_tool.cc:1074] tablet ca4f195a6dd14be981395538c2a33ad9: '10b47782558c41ee91f5e164a372049a' -> 'bceeae1c79964acabdae8ff771bf0c97' move scheduled
I20240501 20:10:07.679615 7475 rebalancer_tool.cc:1074] tablet 2aa1a5f266e840698f2127289628dcba: '1d34e18a1d394d3f91c9f50d1c7bea4e' -> 'b9957d4086bc4a76b18ed9265a5739d5' move scheduled
I20240501 20:10:08.939201 7475 rebalancer_tool.cc:1136] tablet bb85d7825c564339bd554deeded45cec: 'bceeae1c79964acabdae8ff771bf0c97' -> '1d34e18a1d394d3f91c9f50d1c7bea4e' move completed: OK
I20240501 20:10:08.952404 7475 rebalancer_tool.cc:1136] tablet 9d634c1feb7449f686cf3af2a07181e0: '10b47782558c41ee91f5e164a372049a' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move completed: OK
I20240501 20:10:08.966156 7475 rebalancer_tool.cc:1136] tablet dfaa3794205d482c90aa4f860fa99cf3: '1d34e18a1d394d3f91c9f50d1c7bea4e' -> 'bceeae1c79964acabdae8ff771bf0c97' move completed: OK
I20240501 20:10:08.994992 7475 rebalancer_tool.cc:1136] tablet 9d165c29dabe4ae3be11ed84702c3cb2: '1d34e18a1d394d3f91c9f50d1c7bea4e' -> 'db0742ee3860468590eefef6ae6074fd' move completed: OK
I20240501 20:10:08.997732 7475 rebalancer_tool.cc:1123] tablet 66c533992f8b4c7a937e79f397636bcc: 'dca57823003e4729b769ca12367d7530' -> 'db0742ee3860468590eefef6ae6074fd' move is abandoned: Not found: 66c533992f8b4c7a937e79f397636bcc: tablet not found
I20240501 20:10:09.012997 7475 rebalancer_tool.cc:1136] tablet c31148e9e4dd42c3a11787d9384c79c2: '3e974c03d3c546f9ab6c91e762a5426b' -> 'b9957d4086bc4a76b18ed9265a5739d5' move completed: OK
I20240501 20:10:09.024495 7475 rebalancer_tool.cc:1136] tablet f40b062e103d4e54a913fa0f4111f623: 'dca57823003e4729b769ca12367d7530' -> 'bceeae1c79964acabdae8ff771bf0c97' move completed: OK
I20240501 20:10:09.059160 7475 rebalancer_tool.cc:1136] tablet 34cb2f3c90cc4158a61f426c0e5cb86b: '3e974c03d3c546f9ab6c91e762a5426b' -> 'b9957d4086bc4a76b18ed9265a5739d5' move completed: OK
I20240501 20:10:09.074827 7475 rebalancer_tool.cc:1136] tablet c66514deec5744da832bc030a582c46a: '1d34e18a1d394d3f91c9f50d1c7bea4e' -> 'bceeae1c79964acabdae8ff771bf0c97' move completed: OK
I20240501 20:10:09.089176 7475 rebalancer_tool.cc:1136] tablet 46df0faa681b469abe9e383b181a6053: '10b47782558c41ee91f5e164a372049a' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move completed: OK
I20240501 20:10:09.136982 7475 rebalancer_tool.cc:1136] tablet ef7c0a48f11945d3953f3461d06bc184: '10b47782558c41ee91f5e164a372049a' -> 'b9957d4086bc4a76b18ed9265a5739d5' move completed: OK
I20240501 20:10:09.146843 7475 rebalancer_tool.cc:1136] tablet 7725fa9f98e9439797a532ed6f2f7dba: '3e974c03d3c546f9ab6c91e762a5426b' -> 'db0742ee3860468590eefef6ae6074fd' move completed: OK
I20240501 20:10:09.230489 7475 rebalancer_tool.cc:1136] tablet fcff4be357084bf0903cfb1c0f62a866: 'dca57823003e4729b769ca12367d7530' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move completed: OK
I20240501 20:10:09.242043 7475 rebalancer_tool.cc:1136] tablet d690199665174c7d9bdbd54d2af4ac70: '10b47782558c41ee91f5e164a372049a' -> 'b9957d4086bc4a76b18ed9265a5739d5' move completed: OK
I20240501 20:10:09.257472 7475 rebalancer_tool.cc:1136] tablet 41d6e6dce35e4d0da594f147a7f9bfc4: '3e974c03d3c546f9ab6c91e762a5426b' -> 'db0742ee3860468590eefef6ae6074fd' move completed: OK
I20240501 20:10:09.362850 7475 rebalancer_tool.cc:1074] tablet 3809cc8ffe6b4a9e99ebbb9d26a65279: '10b47782558c41ee91f5e164a372049a' -> '1d34e18a1d394d3f91c9f50d1c7bea4e' move scheduled
I20240501 20:10:09.528618 7475 rebalancer_tool.cc:1074] tablet 6f2576acd7d34ec1ad27ab04a863df10: 'bceeae1c79964acabdae8ff771bf0c97' -> '3e974c03d3c546f9ab6c91e762a5426b' move scheduled
I20240501 20:10:09.693696 7475 rebalancer_tool.cc:1074] tablet ffaf0a0105ea4e1f9ed9d90c221ebc8a: '1d34e18a1d394d3f91c9f50d1c7bea4e' -> 'b9957d4086bc4a76b18ed9265a5739d5' move scheduled
I20240501 20:10:10.047204 7475 rebalancer_tool.cc:1074] tablet d1291a2b3dcc4c55a26df6a5cc390050: 'db0742ee3860468590eefef6ae6074fd' -> '10b47782558c41ee91f5e164a372049a' move scheduled
I20240501 20:10:10.310397 7475 rebalancer_tool.cc:1074] tablet f18af954cefa48d4882313fe18abbd27: 'dca57823003e4729b769ca12367d7530' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move scheduled
I20240501 20:10:10.644454 7475 rebalancer_tool.cc:1074] tablet f44675b6874a423398de361c7ccfd583: 'bceeae1c79964acabdae8ff771bf0c97' -> '3e974c03d3c546f9ab6c91e762a5426b' move scheduled
I20240501 20:10:10.858834 7475 rebalancer_tool.cc:1074] tablet 4f99f5d9d4bf44c39188c8b5d5362b98: '3e974c03d3c546f9ab6c91e762a5426b' -> 'b9957d4086bc4a76b18ed9265a5739d5' move scheduled
I20240501 20:10:10.981616 7475 rebalancer_tool.cc:1074] tablet 56cb0964dabd41079d4428787efc9a9d: '10b47782558c41ee91f5e164a372049a' -> 'db0742ee3860468590eefef6ae6074fd' move scheduled
I20240501 20:10:11.121170 7475 rebalancer_tool.cc:1074] tablet 7bcb854a8f7545cc86323abd69c26c23: '1d34e18a1d394d3f91c9f50d1c7bea4e' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move scheduled
I20240501 20:10:11.571628 7475 rebalancer_tool.cc:1074] tablet 5215e3f2040747f29aa1c9181a628096: 'ef0a4ada548446b3a531cb39a3c4e38d' -> 'dca57823003e4729b769ca12367d7530' move scheduled
I20240501 20:10:11.784711 7475 rebalancer_tool.cc:1074] tablet 53be6ced1df34676b1a34bc5cfe1ac87: 'bceeae1c79964acabdae8ff771bf0c97' -> '3e974c03d3c546f9ab6c91e762a5426b' move scheduled
I20240501 20:10:13.074303 7475 rebalancer_tool.cc:1136] tablet d1291a2b3dcc4c55a26df6a5cc390050: 'db0742ee3860468590eefef6ae6074fd' -> '10b47782558c41ee91f5e164a372049a' move completed: OK
I20240501 20:10:13.325202 7475 rebalancer_tool.cc:1136] tablet ffaf0a0105ea4e1f9ed9d90c221ebc8a: '1d34e18a1d394d3f91c9f50d1c7bea4e' -> 'b9957d4086bc4a76b18ed9265a5739d5' move completed: OK
I20240501 20:10:13.441051 7475 rebalancer_tool.cc:1136] tablet 3809cc8ffe6b4a9e99ebbb9d26a65279: '10b47782558c41ee91f5e164a372049a' -> '1d34e18a1d394d3f91c9f50d1c7bea4e' move completed: OK
I20240501 20:10:13.502329 7475 rebalancer_tool.cc:1136] tablet 6f2576acd7d34ec1ad27ab04a863df10: 'bceeae1c79964acabdae8ff771bf0c97' -> '3e974c03d3c546f9ab6c91e762a5426b' move completed: OK
I20240501 20:10:13.558306 7475 rebalancer_tool.cc:1136] tablet 2aa1a5f266e840698f2127289628dcba: '1d34e18a1d394d3f91c9f50d1c7bea4e' -> 'b9957d4086bc4a76b18ed9265a5739d5' move completed: OK
I20240501 20:10:13.570809 7475 rebalancer_tool.cc:1136] tablet ca4f195a6dd14be981395538c2a33ad9: '10b47782558c41ee91f5e164a372049a' -> 'bceeae1c79964acabdae8ff771bf0c97' move completed: OK
I20240501 20:10:13.587427 7475 rebalancer_tool.cc:1136] tablet f44675b6874a423398de361c7ccfd583: 'bceeae1c79964acabdae8ff771bf0c97' -> '3e974c03d3c546f9ab6c91e762a5426b' move completed: OK
I20240501 20:10:13.765199 7475 rebalancer_tool.cc:1136] tablet 1c6c89fa7e6f485e889a5197da4bd135: 'dca57823003e4729b769ca12367d7530' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move completed: OK
I20240501 20:10:13.865305 7475 rebalancer_tool.cc:1136] tablet 644d4d293e8c462297c6419965bd2c89: '3e974c03d3c546f9ab6c91e762a5426b' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move completed: OK
I20240501 20:10:13.892123 7475 rebalancer_tool.cc:1074] tablet 91a2bf5ddc03431b963ab5da831fd340: '3e974c03d3c546f9ab6c91e762a5426b' -> 'b9957d4086bc4a76b18ed9265a5739d5' move scheduled
I20240501 20:10:13.968132 7475 rebalancer_tool.cc:1074] tablet a20287c2cab44a62b28037a463d4b49b: '10b47782558c41ee91f5e164a372049a' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move scheduled
I20240501 20:10:14.015938 7475 rebalancer_tool.cc:1074] tablet 23eaac7a50354947b6efa26ac5f01ae7: '3e974c03d3c546f9ab6c91e762a5426b' -> 'bceeae1c79964acabdae8ff771bf0c97' move scheduled
I20240501 20:10:14.076269 7475 rebalancer_tool.cc:1074] tablet a8eefa22b6e74b92b9fef036dea46e3b: '3e974c03d3c546f9ab6c91e762a5426b' -> 'db0742ee3860468590eefef6ae6074fd' move scheduled
I20240501 20:10:14.178855 7475 rebalancer_tool.cc:1074] tablet f82a24aabb2b4386b5916653a97e548c: 'ef0a4ada548446b3a531cb39a3c4e38d' -> '10b47782558c41ee91f5e164a372049a' move scheduled
I20240501 20:10:14.476449 7475 rebalancer_tool.cc:1136] tablet 53be6ced1df34676b1a34bc5cfe1ac87: 'bceeae1c79964acabdae8ff771bf0c97' -> '3e974c03d3c546f9ab6c91e762a5426b' move completed: OK
I20240501 20:10:14.575407 7475 rebalancer_tool.cc:1136] tablet 5215e3f2040747f29aa1c9181a628096: 'ef0a4ada548446b3a531cb39a3c4e38d' -> 'dca57823003e4729b769ca12367d7530' move completed: OK
I20240501 20:10:14.592013 7475 rebalancer_tool.cc:1136] tablet 7bcb854a8f7545cc86323abd69c26c23: '1d34e18a1d394d3f91c9f50d1c7bea4e' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move completed: OK
I20240501 20:10:14.821969 7475 rebalancer_tool.cc:1136] tablet 56cb0964dabd41079d4428787efc9a9d: '10b47782558c41ee91f5e164a372049a' -> 'db0742ee3860468590eefef6ae6074fd' move completed: OK
I20240501 20:10:15.027240 7475 rebalancer_tool.cc:1136] tablet f18af954cefa48d4882313fe18abbd27: 'dca57823003e4729b769ca12367d7530' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move completed: OK
I20240501 20:10:15.058080 7475 rebalancer_tool.cc:1074] tablet 65b0048212a745c08ab24ead346bf74e: '3e974c03d3c546f9ab6c91e762a5426b' -> 'bceeae1c79964acabdae8ff771bf0c97' move scheduled
I20240501 20:10:15.917558 7475 rebalancer_tool.cc:1136] tablet 91a2bf5ddc03431b963ab5da831fd340: '3e974c03d3c546f9ab6c91e762a5426b' -> 'b9957d4086bc4a76b18ed9265a5739d5' move completed: OK
I20240501 20:10:16.015313 7475 rebalancer_tool.cc:1136] tablet 23eaac7a50354947b6efa26ac5f01ae7: '3e974c03d3c546f9ab6c91e762a5426b' -> 'bceeae1c79964acabdae8ff771bf0c97' move completed: OK
I20240501 20:10:16.048290 7475 rebalancer_tool.cc:1136] tablet a20287c2cab44a62b28037a463d4b49b: '10b47782558c41ee91f5e164a372049a' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move completed: OK
I20240501 20:10:16.147442 7475 rebalancer_tool.cc:1074] tablet ddcb6f5090f143c2b672e5630230878f: '3e974c03d3c546f9ab6c91e762a5426b' -> 'b9957d4086bc4a76b18ed9265a5739d5' move scheduled
I20240501 20:10:16.280786 7475 rebalancer_tool.cc:1074] tablet 0bf6bcf89d444b339612d731af604d0a: '3e974c03d3c546f9ab6c91e762a5426b' -> 'bceeae1c79964acabdae8ff771bf0c97' move scheduled
I20240501 20:10:16.577981 7475 rebalancer_tool.cc:1136] tablet f82a24aabb2b4386b5916653a97e548c: 'ef0a4ada548446b3a531cb39a3c4e38d' -> '10b47782558c41ee91f5e164a372049a' move completed: OK
I20240501 20:10:16.722689 7475 rebalancer_tool.cc:1136] tablet a8eefa22b6e74b92b9fef036dea46e3b: '3e974c03d3c546f9ab6c91e762a5426b' -> 'db0742ee3860468590eefef6ae6074fd' move completed: OK
I20240501 20:10:16.888916 7475 rebalancer_tool.cc:1136] tablet 4f99f5d9d4bf44c39188c8b5d5362b98: '3e974c03d3c546f9ab6c91e762a5426b' -> 'b9957d4086bc4a76b18ed9265a5739d5' move completed: OK
I20240501 20:10:16.952806 7475 rebalancer_tool.cc:1074] tablet 35c843bebb484f6cb655c146a02261ba: 'ef0a4ada548446b3a531cb39a3c4e38d' -> '3e974c03d3c546f9ab6c91e762a5426b' move scheduled
I20240501 20:10:17.042367 7475 rebalancer_tool.cc:1074] tablet bb5700eb47e044c182c33183ebb27908: '3e974c03d3c546f9ab6c91e762a5426b' -> 'db0742ee3860468590eefef6ae6074fd' move scheduled
I20240501 20:10:17.969862 7475 rebalancer_tool.cc:1136] tablet 0bf6bcf89d444b339612d731af604d0a: '3e974c03d3c546f9ab6c91e762a5426b' -> 'bceeae1c79964acabdae8ff771bf0c97' move completed: OK
I20240501 20:10:18.022842 7475 rebalancer_tool.cc:1136] tablet ddcb6f5090f143c2b672e5630230878f: '3e974c03d3c546f9ab6c91e762a5426b' -> 'b9957d4086bc4a76b18ed9265a5739d5' move completed: OK
I20240501 20:10:18.645547 7475 rebalancer_tool.cc:1136] tablet 35c843bebb484f6cb655c146a02261ba: 'ef0a4ada548446b3a531cb39a3c4e38d' -> '3e974c03d3c546f9ab6c91e762a5426b' move completed: OK
I20240501 20:10:18.712466 7475 rebalancer_tool.cc:1136] tablet 65b0048212a745c08ab24ead346bf74e: '3e974c03d3c546f9ab6c91e762a5426b' -> 'bceeae1c79964acabdae8ff771bf0c97' move completed: OK
I20240501 20:10:18.736763 7475 rebalancer_tool.cc:1136] tablet bb5700eb47e044c182c33183ebb27908: '3e974c03d3c546f9ab6c91e762a5426b' -> 'db0742ee3860468590eefef6ae6074fd' move completed: OK
W20240501 20:10:19.825099 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table affa3c54d7ce469c819876e9e5aedb60: could not find any suitable replica to move from server ef0a4ada548446b3a531cb39a3c4e38d to server 1d34e18a1d394d3f91c9f50d1c7bea4e
W20240501 20:10:19.825743 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b43133d24f1646bca3476aba5ea45126: could not find any suitable replica to move from server 3e974c03d3c546f9ab6c91e762a5426b to server dca57823003e4729b769ca12367d7530
I20240501 20:10:19.854539 7475 rebalancer_tool.cc:1074] tablet 264d3fcd3fae4acd8d76306c0f32ed72: 'db0742ee3860468590eefef6ae6074fd' -> '3e974c03d3c546f9ab6c91e762a5426b' move scheduled
I20240501 20:10:19.890818 7475 rebalancer_tool.cc:1074] tablet dfaa3794205d482c90aa4f860fa99cf3: 'dca57823003e4729b769ca12367d7530' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move scheduled
I20240501 20:10:19.950600 7475 rebalancer_tool.cc:1074] tablet 129322eab07149d2a6fb7eef96100e7a: 'bceeae1c79964acabdae8ff771bf0c97' -> '1d34e18a1d394d3f91c9f50d1c7bea4e' move scheduled
I20240501 20:10:20.047258 7475 rebalancer_tool.cc:1074] tablet c1e7ac38dc9c44d29e2d8127904407eb: 'b9957d4086bc4a76b18ed9265a5739d5' -> '10b47782558c41ee91f5e164a372049a' move scheduled
I20240501 20:10:20.212762 7475 rebalancer_tool.cc:1074] tablet 79af181b5f014a40a4c123b47a720dec: 'db0742ee3860468590eefef6ae6074fd' -> '3e974c03d3c546f9ab6c91e762a5426b' move scheduled
I20240501 20:10:20.458961 7475 rebalancer_tool.cc:1074] tablet 5a7b2f01b5684df5800987591a390a17: 'ef0a4ada548446b3a531cb39a3c4e38d' -> 'dca57823003e4729b769ca12367d7530' move scheduled
I20240501 20:10:20.661916 7475 rebalancer_tool.cc:1074] tablet 9e31f8d1809244efae05ab9254c163ce: 'bceeae1c79964acabdae8ff771bf0c97' -> '1d34e18a1d394d3f91c9f50d1c7bea4e' move scheduled
I20240501 20:10:20.825188 7475 rebalancer_tool.cc:1074] tablet 61af956bc72b4e3e994a7daf58d1a699: 'b9957d4086bc4a76b18ed9265a5739d5' -> '10b47782558c41ee91f5e164a372049a' move scheduled
I20240501 20:10:20.998154 7475 rebalancer_tool.cc:1074] tablet c66514deec5744da832bc030a582c46a: '3e974c03d3c546f9ab6c91e762a5426b' -> 'db0742ee3860468590eefef6ae6074fd' move scheduled
I20240501 20:10:21.445079 7475 rebalancer_tool.cc:1074] tablet e7f35eafb0be4f4eab73133917b22bbf: 'ef0a4ada548446b3a531cb39a3c4e38d' -> 'dca57823003e4729b769ca12367d7530' move scheduled
I20240501 20:10:22.334982 7475 rebalancer_tool.cc:1074] tablet 41d6e6dce35e4d0da594f147a7f9bfc4: '10b47782558c41ee91f5e164a372049a' -> 'bceeae1c79964acabdae8ff771bf0c97' move scheduled
I20240501 20:10:22.561020 7475 rebalancer_tool.cc:1074] tablet 7afb8e3154a64be78b8041b137b3681a: '1d34e18a1d394d3f91c9f50d1c7bea4e' -> '3e974c03d3c546f9ab6c91e762a5426b' move scheduled
I20240501 20:10:22.617105 7475 rebalancer_tool.cc:1074] tablet e84a2462cb7e4e559fba887790a65a1f: 'db0742ee3860468590eefef6ae6074fd' -> 'b9957d4086bc4a76b18ed9265a5739d5' move scheduled
I20240501 20:10:22.722370 7475 rebalancer_tool.cc:1074] tablet 030335599b754c81810fe0d139ef51f1: 'ef0a4ada548446b3a531cb39a3c4e38d' -> '10b47782558c41ee91f5e164a372049a' move scheduled
I20240501 20:10:22.993141 7475 rebalancer_tool.cc:1074] tablet 7725fa9f98e9439797a532ed6f2f7dba: 'dca57823003e4729b769ca12367d7530' -> 'b9957d4086bc4a76b18ed9265a5739d5' move scheduled
I20240501 20:10:23.293044 7475 rebalancer_tool.cc:1074] tablet 3f63c61d85524d87a4b911c321f486c4: 'bceeae1c79964acabdae8ff771bf0c97' -> '3e974c03d3c546f9ab6c91e762a5426b' move scheduled
I20240501 20:10:23.631644 7475 rebalancer_tool.cc:1074] tablet c147c98b674e499992f0b9f556ac6815: 'ef0a4ada548446b3a531cb39a3c4e38d' -> '1d34e18a1d394d3f91c9f50d1c7bea4e' move scheduled
I20240501 20:10:23.782557 7475 rebalancer_tool.cc:1074] tablet 6cb1ca3dbf614f94b39e8ddd28ab04fc: 'db0742ee3860468590eefef6ae6074fd' -> '10b47782558c41ee91f5e164a372049a' move scheduled
I20240501 20:10:23.939790 7475 rebalancer_tool.cc:1074] tablet 743d7fd2ae014352a5681c2a8ee4447d: 'b9957d4086bc4a76b18ed9265a5739d5' -> 'dca57823003e4729b769ca12367d7530' move scheduled
I20240501 20:10:25.059881 7475 rebalancer_tool.cc:1136] tablet 030335599b754c81810fe0d139ef51f1: 'ef0a4ada548446b3a531cb39a3c4e38d' -> '10b47782558c41ee91f5e164a372049a' move completed: OK
I20240501 20:10:25.749280 7475 rebalancer_tool.cc:1136] tablet 7725fa9f98e9439797a532ed6f2f7dba: 'dca57823003e4729b769ca12367d7530' -> 'b9957d4086bc4a76b18ed9265a5739d5' move completed: OK
I20240501 20:10:26.005234 7475 rebalancer_tool.cc:1136] tablet e84a2462cb7e4e559fba887790a65a1f: 'db0742ee3860468590eefef6ae6074fd' -> 'b9957d4086bc4a76b18ed9265a5739d5' move completed: OK
I20240501 20:10:26.143631 7475 rebalancer_tool.cc:1136] tablet 7afb8e3154a64be78b8041b137b3681a: '1d34e18a1d394d3f91c9f50d1c7bea4e' -> '3e974c03d3c546f9ab6c91e762a5426b' move completed: OK
I20240501 20:10:26.194464 7475 rebalancer_tool.cc:1136] tablet 41d6e6dce35e4d0da594f147a7f9bfc4: '10b47782558c41ee91f5e164a372049a' -> 'bceeae1c79964acabdae8ff771bf0c97' move completed: OK
I20240501 20:10:26.248844 7475 rebalancer_tool.cc:1136] tablet c66514deec5744da832bc030a582c46a: '3e974c03d3c546f9ab6c91e762a5426b' -> 'db0742ee3860468590eefef6ae6074fd' move completed: OK
I20240501 20:10:26.258260 7475 rebalancer_tool.cc:1136] tablet 61af956bc72b4e3e994a7daf58d1a699: 'b9957d4086bc4a76b18ed9265a5739d5' -> '10b47782558c41ee91f5e164a372049a' move completed: OK
I20240501 20:10:26.269194 7475 rebalancer_tool.cc:1136] tablet 9e31f8d1809244efae05ab9254c163ce: 'bceeae1c79964acabdae8ff771bf0c97' -> '1d34e18a1d394d3f91c9f50d1c7bea4e' move completed: OK
I20240501 20:10:26.282047 7475 rebalancer_tool.cc:1136] tablet 5a7b2f01b5684df5800987591a390a17: 'ef0a4ada548446b3a531cb39a3c4e38d' -> 'dca57823003e4729b769ca12367d7530' move completed: OK
I20240501 20:10:26.296480 7475 rebalancer_tool.cc:1136] tablet 79af181b5f014a40a4c123b47a720dec: 'db0742ee3860468590eefef6ae6074fd' -> '3e974c03d3c546f9ab6c91e762a5426b' move completed: OK
I20240501 20:10:26.404538 7475 rebalancer_tool.cc:1136] tablet c1e7ac38dc9c44d29e2d8127904407eb: 'b9957d4086bc4a76b18ed9265a5739d5' -> '10b47782558c41ee91f5e164a372049a' move completed: OK
I20240501 20:10:26.468391 7475 rebalancer_tool.cc:1136] tablet e7f35eafb0be4f4eab73133917b22bbf: 'ef0a4ada548446b3a531cb39a3c4e38d' -> 'dca57823003e4729b769ca12367d7530' move completed: OK
I20240501 20:10:26.479810 7475 rebalancer_tool.cc:1136] tablet 129322eab07149d2a6fb7eef96100e7a: 'bceeae1c79964acabdae8ff771bf0c97' -> '1d34e18a1d394d3f91c9f50d1c7bea4e' move completed: OK
I20240501 20:10:26.519598 7475 rebalancer_tool.cc:1136] tablet 264d3fcd3fae4acd8d76306c0f32ed72: 'db0742ee3860468590eefef6ae6074fd' -> '3e974c03d3c546f9ab6c91e762a5426b' move completed: OK
I20240501 20:10:26.549064 7475 rebalancer_tool.cc:1074] tablet d13178f10db0435891b97cebaed6dd5d: 'b9957d4086bc4a76b18ed9265a5739d5' -> '3e974c03d3c546f9ab6c91e762a5426b' move scheduled
I20240501 20:10:26.582995 7475 rebalancer_tool.cc:1074] tablet c5b4fbeda51f40b8ac54baf5fb241710: '1d34e18a1d394d3f91c9f50d1c7bea4e' -> '10b47782558c41ee91f5e164a372049a' move scheduled
I20240501 20:10:26.676789 7475 rebalancer_tool.cc:1074] tablet 797b2374d4214c8eb39982c339a7fbad: 'bceeae1c79964acabdae8ff771bf0c97' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move scheduled
I20240501 20:10:26.911091 7475 rebalancer_tool.cc:1074] tablet 26ed0760e9074bd896e55e99756602bb: 'db0742ee3860468590eefef6ae6074fd' -> 'dca57823003e4729b769ca12367d7530' move scheduled
I20240501 20:10:26.973564 7475 rebalancer_tool.cc:1074] tablet 48528ec242974de4ac20eb0161f0cb05: 'b9957d4086bc4a76b18ed9265a5739d5' -> '3e974c03d3c546f9ab6c91e762a5426b' move scheduled
I20240501 20:10:27.327150 7475 rebalancer_tool.cc:1074] tablet 00ae88161b514e4aa4b8d134a19de001: 'ef0a4ada548446b3a531cb39a3c4e38d' -> '1d34e18a1d394d3f91c9f50d1c7bea4e' move scheduled
I20240501 20:10:27.575973 7475 rebalancer_tool.cc:1074] tablet 65b0048212a745c08ab24ead346bf74e: '10b47782558c41ee91f5e164a372049a' -> 'db0742ee3860468590eefef6ae6074fd' move scheduled
I20240501 20:10:27.840289 7475 rebalancer_tool.cc:1074] tablet 5215e3f2040747f29aa1c9181a628096: 'bceeae1c79964acabdae8ff771bf0c97' -> '3e974c03d3c546f9ab6c91e762a5426b' move scheduled
I20240501 20:10:28.115270 7475 rebalancer_tool.cc:1074] tablet ca4f195a6dd14be981395538c2a33ad9: 'dca57823003e4729b769ca12367d7530' -> 'db0742ee3860468590eefef6ae6074fd' move scheduled
I20240501 20:10:28.349157 7475 rebalancer_tool.cc:1074] tablet 2aa1a5f266e840698f2127289628dcba: '10b47782558c41ee91f5e164a372049a' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move scheduled
I20240501 20:10:28.967694 7475 rebalancer_tool.cc:1074] tablet b2619f3bed304d3ab864f56d5b655574: 'bceeae1c79964acabdae8ff771bf0c97' -> '3e974c03d3c546f9ab6c91e762a5426b' move scheduled
I20240501 20:10:29.300945 7475 rebalancer_tool.cc:1074] tablet 1d860bb204134f0994462275519feecf: 'db0742ee3860468590eefef6ae6074fd' -> 'dca57823003e4729b769ca12367d7530' move scheduled
I20240501 20:10:29.537708 7475 rebalancer_tool.cc:1136] tablet 00ae88161b514e4aa4b8d134a19de001: 'ef0a4ada548446b3a531cb39a3c4e38d' -> '1d34e18a1d394d3f91c9f50d1c7bea4e' move completed: OK
I20240501 20:10:29.640684 7475 rebalancer_tool.cc:1136] tablet 26ed0760e9074bd896e55e99756602bb: 'db0742ee3860468590eefef6ae6074fd' -> 'dca57823003e4729b769ca12367d7530' move completed: OK
I20240501 20:10:29.735237 7475 rebalancer_tool.cc:1136] tablet 797b2374d4214c8eb39982c339a7fbad: 'bceeae1c79964acabdae8ff771bf0c97' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move completed: OK
I20240501 20:10:29.850522 7475 rebalancer_tool.cc:1136] tablet 48528ec242974de4ac20eb0161f0cb05: 'b9957d4086bc4a76b18ed9265a5739d5' -> '3e974c03d3c546f9ab6c91e762a5426b' move completed: OK
I20240501 20:10:29.948129 7475 rebalancer_tool.cc:1136] tablet c5b4fbeda51f40b8ac54baf5fb241710: '1d34e18a1d394d3f91c9f50d1c7bea4e' -> '10b47782558c41ee91f5e164a372049a' move completed: OK
I20240501 20:10:29.980561 7475 rebalancer_tool.cc:1136] tablet d13178f10db0435891b97cebaed6dd5d: 'b9957d4086bc4a76b18ed9265a5739d5' -> '3e974c03d3c546f9ab6c91e762a5426b' move completed: OK
I20240501 20:10:29.997954 7475 rebalancer_tool.cc:1136] tablet 743d7fd2ae014352a5681c2a8ee4447d: 'b9957d4086bc4a76b18ed9265a5739d5' -> 'dca57823003e4729b769ca12367d7530' move completed: OK
I20240501 20:10:30.018123 7475 rebalancer_tool.cc:1136] tablet 6cb1ca3dbf614f94b39e8ddd28ab04fc: 'db0742ee3860468590eefef6ae6074fd' -> '10b47782558c41ee91f5e164a372049a' move completed: OK
I20240501 20:10:30.149296 7475 rebalancer_tool.cc:1136] tablet c147c98b674e499992f0b9f556ac6815: 'ef0a4ada548446b3a531cb39a3c4e38d' -> '1d34e18a1d394d3f91c9f50d1c7bea4e' move completed: OK
I20240501 20:10:30.163442 7475 rebalancer_tool.cc:1136] tablet 3f63c61d85524d87a4b911c321f486c4: 'bceeae1c79964acabdae8ff771bf0c97' -> '3e974c03d3c546f9ab6c91e762a5426b' move completed: OK
I20240501 20:10:30.185699 7475 rebalancer_tool.cc:1136] tablet dfaa3794205d482c90aa4f860fa99cf3: 'dca57823003e4729b769ca12367d7530' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move completed: OK
I20240501 20:10:30.226460 7475 rebalancer_tool.cc:1074] tablet 965b65ec896648698fbf3e1266d8e678: 'ef0a4ada548446b3a531cb39a3c4e38d' -> '10b47782558c41ee91f5e164a372049a' move scheduled
I20240501 20:10:30.364349 7475 rebalancer_tool.cc:1074] tablet ffaf0a0105ea4e1f9ed9d90c221ebc8a: '3e974c03d3c546f9ab6c91e762a5426b' -> 'bceeae1c79964acabdae8ff771bf0c97' move scheduled
I20240501 20:10:30.419289 7475 rebalancer_tool.cc:1074] tablet f40b062e103d4e54a913fa0f4111f623: '3e974c03d3c546f9ab6c91e762a5426b' -> 'dca57823003e4729b769ca12367d7530' move scheduled
I20240501 20:10:30.491863 7475 rebalancer_tool.cc:1074] tablet 25f01a01106f47c489e252c928cf28bb: 'db0742ee3860468590eefef6ae6074fd' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move scheduled
I20240501 20:10:30.945593 7475 rebalancer_tool.cc:1074] tablet 537b5ca846b5485ca1d98351d9038430: 'db0742ee3860468590eefef6ae6074fd' -> '10b47782558c41ee91f5e164a372049a' move scheduled
I20240501 20:10:31.059011 7475 rebalancer_tool.cc:1074] tablet 46df0faa681b469abe9e383b181a6053: '3e974c03d3c546f9ab6c91e762a5426b' -> 'bceeae1c79964acabdae8ff771bf0c97' move scheduled
I20240501 20:10:31.184888 7475 rebalancer_tool.cc:1136] tablet 1d860bb204134f0994462275519feecf: 'db0742ee3860468590eefef6ae6074fd' -> 'dca57823003e4729b769ca12367d7530' move completed: OK
I20240501 20:10:31.196022 7475 rebalancer_tool.cc:1136] tablet ca4f195a6dd14be981395538c2a33ad9: 'dca57823003e4729b769ca12367d7530' -> 'db0742ee3860468590eefef6ae6074fd' move completed: OK
I20240501 20:10:31.210438 7475 rebalancer_tool.cc:1136] tablet 65b0048212a745c08ab24ead346bf74e: '10b47782558c41ee91f5e164a372049a' -> 'db0742ee3860468590eefef6ae6074fd' move completed: OK
I20240501 20:10:31.251094 7475 rebalancer_tool.cc:1136] tablet 5215e3f2040747f29aa1c9181a628096: 'bceeae1c79964acabdae8ff771bf0c97' -> '3e974c03d3c546f9ab6c91e762a5426b' move completed: OK
I20240501 20:10:31.272274 7475 rebalancer_tool.cc:1136] tablet 2aa1a5f266e840698f2127289628dcba: '10b47782558c41ee91f5e164a372049a' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move completed: OK
I20240501 20:10:31.284731 7475 rebalancer_tool.cc:1136] tablet b2619f3bed304d3ab864f56d5b655574: 'bceeae1c79964acabdae8ff771bf0c97' -> '3e974c03d3c546f9ab6c91e762a5426b' move completed: OK
I20240501 20:10:31.356226 7475 rebalancer_tool.cc:1074] tablet 04a5fb2b808d466491144e64eb54d0fd: 'db0742ee3860468590eefef6ae6074fd' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move scheduled
I20240501 20:10:31.400381 7475 rebalancer_tool.cc:1074] tablet d690199665174c7d9bdbd54d2af4ac70: '3e974c03d3c546f9ab6c91e762a5426b' -> 'db0742ee3860468590eefef6ae6074fd' move scheduled
I20240501 20:10:31.921511 7475 rebalancer_tool.cc:1136] tablet 965b65ec896648698fbf3e1266d8e678: 'ef0a4ada548446b3a531cb39a3c4e38d' -> '10b47782558c41ee91f5e164a372049a' move completed: OK
I20240501 20:10:32.220857 7475 rebalancer_tool.cc:1136] tablet 25f01a01106f47c489e252c928cf28bb: 'db0742ee3860468590eefef6ae6074fd' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move completed: OK
I20240501 20:10:32.268738 7475 rebalancer_tool.cc:1136] tablet f40b062e103d4e54a913fa0f4111f623: '3e974c03d3c546f9ab6c91e762a5426b' -> 'dca57823003e4729b769ca12367d7530' move completed: OK
I20240501 20:10:32.626724 7475 rebalancer_tool.cc:1136] tablet 537b5ca846b5485ca1d98351d9038430: 'db0742ee3860468590eefef6ae6074fd' -> '10b47782558c41ee91f5e164a372049a' move completed: OK
I20240501 20:10:33.069157 7475 rebalancer_tool.cc:1136] tablet d690199665174c7d9bdbd54d2af4ac70: '3e974c03d3c546f9ab6c91e762a5426b' -> 'db0742ee3860468590eefef6ae6074fd' move completed: OK
I20240501 20:10:33.154422 7475 rebalancer_tool.cc:1136] tablet 04a5fb2b808d466491144e64eb54d0fd: 'db0742ee3860468590eefef6ae6074fd' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move completed: OK
I20240501 20:10:33.263748 7475 rebalancer_tool.cc:1136] tablet ffaf0a0105ea4e1f9ed9d90c221ebc8a: '3e974c03d3c546f9ab6c91e762a5426b' -> 'bceeae1c79964acabdae8ff771bf0c97' move completed: OK
I20240501 20:10:34.152254 7475 rebalancer_tool.cc:1136] tablet 46df0faa681b469abe9e383b181a6053: '3e974c03d3c546f9ab6c91e762a5426b' -> 'bceeae1c79964acabdae8ff771bf0c97' move completed: OK
W20240501 20:10:34.829994 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4a624bfe1b6b4d6b8ab2c2c578c11688: could not find any suitable replica to move from server 3e974c03d3c546f9ab6c91e762a5426b to server dca57823003e4729b769ca12367d7530
W20240501 20:10:34.830222 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 19fe11bc7ef54ec8be09ca4882586266: could not find any suitable replica to move from server db0742ee3860468590eefef6ae6074fd to server bceeae1c79964acabdae8ff771bf0c97
W20240501 20:10:34.830355 7475 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4966f0b362754500952a1e7f97c65b0b: could not find any suitable replica to move from server 1d34e18a1d394d3f91c9f50d1c7bea4e to server bceeae1c79964acabdae8ff771bf0c97
I20240501 20:10:34.858647 7475 rebalancer_tool.cc:1074] tablet 1d3bb8370e7c41dc9cf7f1c835ac275b: '1d34e18a1d394d3f91c9f50d1c7bea4e' -> 'b9957d4086bc4a76b18ed9265a5739d5' move scheduled
I20240501 20:10:34.893175 7475 rebalancer_tool.cc:1074] tablet 024c925714cb419aaa8870a053a6ca2f: 'bceeae1c79964acabdae8ff771bf0c97' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move scheduled
I20240501 20:10:34.939520 7475 rebalancer_tool.cc:1074] tablet 376ae6034f394a0fb0cab0d8a9241134: 'dca57823003e4729b769ca12367d7530' -> '3e974c03d3c546f9ab6c91e762a5426b' move scheduled
I20240501 20:10:34.994961 7475 rebalancer_tool.cc:1074] tablet 82260307259e4a9fb12231ff4abd79c5: '10b47782558c41ee91f5e164a372049a' -> 'db0742ee3860468590eefef6ae6074fd' move scheduled
I20240501 20:10:35.024983 7475 rebalancer_tool.cc:1074] tablet 786296114e8a47c68df103934ec055d5: 'b9957d4086bc4a76b18ed9265a5739d5' -> '1d34e18a1d394d3f91c9f50d1c7bea4e' move scheduled
I20240501 20:10:35.058097 7475 rebalancer_tool.cc:1074] tablet 92ce2c1ce3e0449bab5490b690721cbf: 'bceeae1c79964acabdae8ff771bf0c97' -> '10b47782558c41ee91f5e164a372049a' move scheduled
I20240501 20:10:35.098701 7475 rebalancer_tool.cc:1074] tablet cd21a0faa3bb487cb80f0951b12b3d67: 'ef0a4ada548446b3a531cb39a3c4e38d' -> 'dca57823003e4729b769ca12367d7530' move scheduled
I20240501 20:10:35.157083 7475 rebalancer_tool.cc:1074] tablet ce0b97a1badb4935a4553902e6feb20b: '3e974c03d3c546f9ab6c91e762a5426b' -> 'db0742ee3860468590eefef6ae6074fd' move scheduled
I20240501 20:10:35.221684 7475 rebalancer_tool.cc:1074] tablet 9709a8bb107742b79078f91da9fab8f6: 'b9957d4086bc4a76b18ed9265a5739d5' -> '10b47782558c41ee91f5e164a372049a' move scheduled
I20240501 20:10:35.276119 7475 rebalancer_tool.cc:1074] tablet 9e0ecc1043af4bc1ab31dfa80ea47bb4: 'bceeae1c79964acabdae8ff771bf0c97' -> '1d34e18a1d394d3f91c9f50d1c7bea4e' move scheduled
I20240501 20:10:35.336659 7475 rebalancer_tool.cc:1074] tablet 99bd0e9f4cd7496785ad71480d7d3d61: 'ef0a4ada548446b3a531cb39a3c4e38d' -> 'b9957d4086bc4a76b18ed9265a5739d5' move scheduled
I20240501 20:10:35.372264 7475 rebalancer_tool.cc:1074] tablet 9c77905da1b3453fa9a83366f7c51c37: '3e974c03d3c546f9ab6c91e762a5426b' -> 'dca57823003e4729b769ca12367d7530' move scheduled
I20240501 20:10:35.467161 7475 rebalancer_tool.cc:1074] tablet 56b2082919b548bfb56b939373caf770: 'db0742ee3860468590eefef6ae6074fd' -> '1d34e18a1d394d3f91c9f50d1c7bea4e' move scheduled
I20240501 20:10:35.527108 7475 rebalancer_tool.cc:1074] tablet 4613582180754f729afc8d0b44356e97: '10b47782558c41ee91f5e164a372049a' -> 'bceeae1c79964acabdae8ff771bf0c97' move scheduled
I20240501 20:10:35.786146 7475 rebalancer_tool.cc:1074] tablet 67703d84deb9417bb21bde6365deda2f: 'b9957d4086bc4a76b18ed9265a5739d5' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move scheduled
I20240501 20:10:35.887131 7475 rebalancer_tool.cc:1074] tablet 3ffe6b742c3743d8b7de7a0c4105637f: 'db0742ee3860468590eefef6ae6074fd' -> '3e974c03d3c546f9ab6c91e762a5426b' move scheduled
I20240501 20:10:36.132169 7475 rebalancer_tool.cc:1074] tablet 79af181b5f014a40a4c123b47a720dec: '10b47782558c41ee91f5e164a372049a' -> '1d34e18a1d394d3f91c9f50d1c7bea4e' move scheduled
W20240501 20:10:37.479933 12253 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:10:37.100004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.219.69:46237 (local address 127.0.0.1:57562)
0501 20:10:37.116148 (+ 16144us) negotiation.cc:107] Waiting for socket to connect
0501 20:10:37.116176 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0501 20:10:37.116346 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 20:10:37.478497 (+362151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":15970,"thread_start_us":18030,"threads_started":1}
I20240501 20:10:37.481153 7475 rebalancer_tool.cc:1123] tablet 99bd0e9f4cd7496785ad71480d7d3d61: 'ef0a4ada548446b3a531cb39a3c4e38d' -> 'b9957d4086bc4a76b18ed9265a5739d5' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:37.509332 7475 rebalancer_tool.cc:1136] tablet 9e0ecc1043af4bc1ab31dfa80ea47bb4: 'bceeae1c79964acabdae8ff771bf0c97' -> '1d34e18a1d394d3f91c9f50d1c7bea4e' move completed: OK
I20240501 20:10:37.582952 7475 rebalancer_tool.cc:1136] tablet 92ce2c1ce3e0449bab5490b690721cbf: 'bceeae1c79964acabdae8ff771bf0c97' -> '10b47782558c41ee91f5e164a372049a' move completed: OK
I20240501 20:10:37.600229 7475 rebalancer_tool.cc:1136] tablet 786296114e8a47c68df103934ec055d5: 'b9957d4086bc4a76b18ed9265a5739d5' -> '1d34e18a1d394d3f91c9f50d1c7bea4e' move completed: OK
I20240501 20:10:37.692711 7475 rebalancer_tool.cc:1136] tablet 82260307259e4a9fb12231ff4abd79c5: '10b47782558c41ee91f5e164a372049a' -> 'db0742ee3860468590eefef6ae6074fd' move completed: OK
I20240501 20:10:37.756080 7475 rebalancer_tool.cc:1123] tablet cd21a0faa3bb487cb80f0951b12b3d67: 'ef0a4ada548446b3a531cb39a3c4e38d' -> 'dca57823003e4729b769ca12367d7530' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:37.797134 7475 rebalancer_tool.cc:1136] tablet 376ae6034f394a0fb0cab0d8a9241134: 'dca57823003e4729b769ca12367d7530' -> '3e974c03d3c546f9ab6c91e762a5426b' move completed: OK
I20240501 20:10:37.808786 7475 rebalancer_tool.cc:1136] tablet 024c925714cb419aaa8870a053a6ca2f: 'bceeae1c79964acabdae8ff771bf0c97' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move completed: OK
I20240501 20:10:37.822736 7475 rebalancer_tool.cc:1136] tablet 1d3bb8370e7c41dc9cf7f1c835ac275b: '1d34e18a1d394d3f91c9f50d1c7bea4e' -> 'b9957d4086bc4a76b18ed9265a5739d5' move completed: OK
I20240501 20:10:37.865381 7475 rebalancer_tool.cc:1074] tablet bec3cae7dc9241fd8eae761967932f6e: '1d34e18a1d394d3f91c9f50d1c7bea4e' -> 'ef0a4ada548446b3a531cb39a3c4e38d' move scheduled
I20240501 20:10:37.896260 7475 rebalancer_tool.cc:1074] tablet aecca0e22b084a8b95c98ef7e7b90b29: 'ef0a4ada548446b3a531cb39a3c4e38d' -> '10b47782558c41ee91f5e164a372049a' move scheduled
I20240501 20:10:37.908931 7475 rebalancer_tool.cc:1084] tablet accaff8513134f96b5502b5c5385d892: 'b9957d4086bc4a76b18ed9265a5739d5' -> 'db0742ee3860468590eefef6ae6074fd' move ignored: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:37.909113 7475 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): unacceptable health status UNAVAILABLE
I20240501 20:10:39.136662 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 4152
I20240501 20:10:39.153270 4274 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:10:39.154633 7222 consensus_peers.cc:485] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:10:39.280529 4434 consensus_peers.cc:485] T a20287c2cab44a62b28037a463d4b49b P 3e974c03d3c546f9ab6c91e762a5426b -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:10:39.321844 12482 ts_tablet_manager.cc:902] T 79af181b5f014a40a4c123b47a720dec P 1d34e18a1d394d3f91c9f50d1c7bea4e: Initiating tablet copy from peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:39.323055 12482 tablet_copy_client.cc:308] T 79af181b5f014a40a4c123b47a720dec P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Beginning tablet copy session from remote peer at address 127.29.219.66:40689
I20240501 20:10:39.323618 4388 tablet_copy_service.cc:140] P 10b47782558c41ee91f5e164a372049a: Received BeginTabletCopySession request for tablet 79af181b5f014a40a4c123b47a720dec from peer 1d34e18a1d394d3f91c9f50d1c7bea4e ({username='slave'} at 127.29.219.68:36333)
I20240501 20:10:39.323773 4388 tablet_copy_service.cc:161] P 10b47782558c41ee91f5e164a372049a: Beginning new tablet copy session on tablet 79af181b5f014a40a4c123b47a720dec from peer 1d34e18a1d394d3f91c9f50d1c7bea4e at {username='slave'} at 127.29.219.68:36333: session id = 1d34e18a1d394d3f91c9f50d1c7bea4e-79af181b5f014a40a4c123b47a720dec
I20240501 20:10:39.326279 4388 tablet_copy_source_session.cc:205] T 79af181b5f014a40a4c123b47a720dec P 10b47782558c41ee91f5e164a372049a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 20:10:39.327610 12482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79af181b5f014a40a4c123b47a720dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:39.332844 12482 tablet_copy_client.cc:783] T 79af181b5f014a40a4c123b47a720dec P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Starting download of 0 data blocks...
I20240501 20:10:39.333091 12482 tablet_copy_client.cc:647] T 79af181b5f014a40a4c123b47a720dec P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Starting download of 1 WAL segments...
I20240501 20:10:39.334513 12482 tablet_copy_client.cc:523] T 79af181b5f014a40a4c123b47a720dec P 1d34e18a1d394d3f91c9f50d1c7bea4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:10:39.336964 12482 tablet_bootstrap.cc:492] T 79af181b5f014a40a4c123b47a720dec P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap starting.
I20240501 20:10:39.343633 12482 tablet_bootstrap.cc:492] T 79af181b5f014a40a4c123b47a720dec P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:10:39.343930 12482 tablet_bootstrap.cc:492] T 79af181b5f014a40a4c123b47a720dec P 1d34e18a1d394d3f91c9f50d1c7bea4e: Bootstrap complete.
I20240501 20:10:39.344128 12482 ts_tablet_manager.cc:1367] T 79af181b5f014a40a4c123b47a720dec P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.002s
I20240501 20:10:39.344758 12482 raft_consensus.cc:348] T 79af181b5f014a40a4c123b47a720dec P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:39.344941 12482 raft_consensus.cc:729] T 79af181b5f014a40a4c123b47a720dec P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 1d34e18a1d394d3f91c9f50d1c7bea4e, State: Initialized, Role: LEARNER
I20240501 20:10:39.345130 12482 consensus_queue.cc:260] T 79af181b5f014a40a4c123b47a720dec P 1d34e18a1d394d3f91c9f50d1c7bea4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { replace: true } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: NON_VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: true } }
I20240501 20:10:39.345693 12482 ts_tablet_manager.cc:1398] T 79af181b5f014a40a4c123b47a720dec P 1d34e18a1d394d3f91c9f50d1c7bea4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 20:10:39.346503 4388 tablet_copy_service.cc:342] P 10b47782558c41ee91f5e164a372049a: Request end of tablet copy session 1d34e18a1d394d3f91c9f50d1c7bea4e-79af181b5f014a40a4c123b47a720dec received from {username='slave'} at 127.29.219.68:36333
I20240501 20:10:39.346644 4388 tablet_copy_service.cc:433] P 10b47782558c41ee91f5e164a372049a: ending tablet copy session 1d34e18a1d394d3f91c9f50d1c7bea4e-79af181b5f014a40a4c123b47a720dec on tablet 79af181b5f014a40a4c123b47a720dec with peer 1d34e18a1d394d3f91c9f50d1c7bea4e
I20240501 20:10:39.402696 12437 raft_consensus.cc:1050] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97: attempting to promote NON_VOTER 10b47782558c41ee91f5e164a372049a to VOTER
I20240501 20:10:39.403364 12437 consensus_queue.cc:237] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } }
I20240501 20:10:39.406175 4495 raft_consensus.cc:1261] T aecca0e22b084a8b95c98ef7e7b90b29 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:39.407084 4368 raft_consensus.cc:1261] T aecca0e22b084a8b95c98ef7e7b90b29 P 10b47782558c41ee91f5e164a372049a [term 1 LEARNER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 20:10:39.407441 11828 consensus_queue.cc:1035] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 20:10:39.409315 7222 consensus_peers.cc:485] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:39.409938 11828 consensus_queue.cc:1035] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 20:10:39.413939 11935 raft_consensus.cc:2932] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:39.414445 4368 raft_consensus.cc:2932] T aecca0e22b084a8b95c98ef7e7b90b29 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:39.423974 4093 catalog_manager.cc:5484] T aecca0e22b084a8b95c98ef7e7b90b29 P 10b47782558c41ee91f5e164a372049a reported cstate change: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:39.428267 4495 raft_consensus.cc:2932] T aecca0e22b084a8b95c98ef7e7b90b29 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 10b47782558c41ee91f5e164a372049a (127.29.219.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:39.439410 7284 consensus_queue.cc:237] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } }
I20240501 20:10:39.441164 4495 raft_consensus.cc:1261] T aecca0e22b084a8b95c98ef7e7b90b29 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:39.441196 4368 raft_consensus.cc:1261] T aecca0e22b084a8b95c98ef7e7b90b29 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Refusing update from remote peer bceeae1c79964acabdae8ff771bf0c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 20:10:39.441594 12437 consensus_queue.cc:1035] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:10:39.443418 12489 consensus_queue.cc:1035] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20240501 20:10:39.444737 4564 consensus_peers.cc:485] T fcff4be357084bf0903cfb1c0f62a866 P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:10:39.444980 11935 raft_consensus.cc:2932] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:39.445549 4498 raft_consensus.cc:2932] T aecca0e22b084a8b95c98ef7e7b90b29 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
I20240501 20:10:39.448696 4077 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aecca0e22b084a8b95c98ef7e7b90b29 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:10:39.451807 4093 catalog_manager.cc:5484] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New cstate: current_term: 1 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:39.455260 4368 raft_consensus.cc:2932] T aecca0e22b084a8b95c98ef7e7b90b29 P 10b47782558c41ee91f5e164a372049a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } } }
W20240501 20:10:39.471083 4434 consensus_peers.cc:485] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:10:39.501767 4434 consensus_peers.cc:485] T 2aa1a5f266e840698f2127289628dcba P 3e974c03d3c546f9ab6c91e762a5426b -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:10:39.543412 7222 consensus_peers.cc:485] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:10:39.544538 4301 consensus_peers.cc:485] T ce89f9b962264db7a91685556f1354a8 P 10b47782558c41ee91f5e164a372049a -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:10:39.548736 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 4152
W20240501 20:10:39.579206 7090 consensus_peers.cc:485] T c1c51d1b44504379819d96ef5aff1edc P db0742ee3860468590eefef6ae6074fd -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:10:39.615726 7222 consensus_peers.cc:485] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:10:39.620247 7355 connection.cc:535] client connection to 127.29.219.65:42941 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:39.620303 7221 connection.cc:535] server connection from 127.29.219.65:33781 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:39.620600 4301 connection.cc:535] client connection to 127.29.219.65:42941 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:39.620692 4300 connection.cc:535] server connection from 127.29.219.65:41491 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:39.622053 4432 connection.cc:535] server connection from 127.29.219.65:36691 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:39.622164 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 4284
W20240501 20:10:39.622220 4564 connection.cc:535] client connection to 127.29.219.65:42941 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:39.622534 12291 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 20:10:39.621243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.219.65:42941 (local address 127.29.219.66:56287)
0501 20:10:39.621598 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0501 20:10:39.621687 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":258}
W20240501 20:10:39.622557 7090 connection.cc:535] client connection to 127.29.219.65:42941 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:39.623332 4301 consensus_peers.cc:485] T f40b062e103d4e54a913fa0f4111f623 P 10b47782558c41ee91f5e164a372049a -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:39.623463 4301 consensus_peers.cc:485] T bb5700eb47e044c182c33183ebb27908 P 10b47782558c41ee91f5e164a372049a -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:39.623522 4301 consensus_peers.cc:485] T 0bf6bcf89d444b339612d731af604d0a P 10b47782558c41ee91f5e164a372049a -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:39.623579 4301 consensus_peers.cc:485] T 91a2bf5ddc03431b963ab5da831fd340 P 10b47782558c41ee91f5e164a372049a -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:39.623629 4301 consensus_peers.cc:485] T 5a7b2f01b5684df5800987591a390a17 P 10b47782558c41ee91f5e164a372049a -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:39.623709 4564 consensus_peers.cc:485] T 743d7fd2ae014352a5681c2a8ee4447d P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:39.623834 4564 consensus_peers.cc:485] T 3ffe6b742c3743d8b7de7a0c4105637f P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:39.624058 7222 connection.cc:535] client connection to 127.29.219.65:42941 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:39.624459 4301 consensus_peers.cc:485] T 26ed0760e9074bd896e55e99756602bb P 10b47782558c41ee91f5e164a372049a -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:39.625644 7090 consensus_peers.cc:485] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:39.625782 7090 consensus_peers.cc:485] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:39.626761 7222 consensus_peers.cc:485] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:39.627126 7355 consensus_peers.cc:485] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:39.627241 7090 consensus_peers.cc:485] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:10:39.627336 7355 consensus_peers.cc:485] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:39.627501 7355 consensus_peers.cc:485] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:39.627609 7355 consensus_peers.cc:485] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:39.635042 4408 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:10:39.653563 7355 consensus_peers.cc:485] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:10:39.653899 7090 consensus_peers.cc:485] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:39.664559 4434 consensus_peers.cc:485] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:39.676877 7355 consensus_peers.cc:485] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:39.697228 7090 consensus_peers.cc:462] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
W20240501 20:10:39.697665 4564 consensus_peers.cc:485] T 25f01a01106f47c489e252c928cf28bb P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:10:39.698266 4564 consensus_peers.cc:485] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:10:39.741096 7090 consensus_peers.cc:485] T 3bdd6de7ac584032aa309269bd0770cf P db0742ee3860468590eefef6ae6074fd -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:10:39.779906 4564 consensus_peers.cc:485] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:10:39.800292 4434 consensus_peers.cc:485] T 1d860bb204134f0994462275519feecf P 3e974c03d3c546f9ab6c91e762a5426b -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:39.818910 7090 consensus_peers.cc:485] T dfb7abcd009341acafded230f877c9f6 P db0742ee3860468590eefef6ae6074fd -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:10:39.842459 4564 consensus_peers.cc:485] T da878c45e7184273ba6d3497e9eef35d P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:10:39.843531 7090 consensus_peers.cc:485] T 797b2374d4214c8eb39982c339a7fbad P db0742ee3860468590eefef6ae6074fd -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:10:39.877893 7090 consensus_peers.cc:485] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:10:39.987215 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 4284
W20240501 20:10:40.042585 7222 connection.cc:535] server connection from 127.29.219.66:32801 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:40.042833 7222 connection.cc:535] client connection to 127.29.219.66:40689 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:40.043177 7355 connection.cc:535] client connection to 127.29.219.66:40689 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:40.043315 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 4416
W20240501 20:10:40.043431 4564 connection.cc:535] client connection to 127.29.219.66:40689 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:40.044045 4434 consensus_peers.cc:485] T 61af956bc72b4e3e994a7daf58d1a699 P 3e974c03d3c546f9ab6c91e762a5426b -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.044174 4434 consensus_peers.cc:485] T 7bcb854a8f7545cc86323abd69c26c23 P 3e974c03d3c546f9ab6c91e762a5426b -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.044842 7090 connection.cc:535] client connection to 127.29.219.66:40689 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:40.045032 4564 consensus_peers.cc:485] T 6f2576acd7d34ec1ad27ab04a863df10 P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.045176 4564 consensus_peers.cc:485] T 04a5fb2b808d466491144e64eb54d0fd P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.045266 4564 consensus_peers.cc:485] T 23eaac7a50354947b6efa26ac5f01ae7 P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.045446 7222 consensus_peers.cc:485] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.045620 7222 consensus_peers.cc:485] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97 -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.045774 7222 consensus_peers.cc:485] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.045925 7222 consensus_peers.cc:485] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.046109 7222 consensus_peers.cc:485] T d1291a2b3dcc4c55a26df6a5cc390050 P bceeae1c79964acabdae8ff771bf0c97 -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.046243 7222 consensus_peers.cc:485] T f82a24aabb2b4386b5916653a97e548c P bceeae1c79964acabdae8ff771bf0c97 -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.047291 7355 consensus_peers.cc:485] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.047426 7355 consensus_peers.cc:485] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.047459 7090 consensus_peers.cc:485] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.047549 7355 consensus_peers.cc:485] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.047570 7090 consensus_peers.cc:485] T 92ce2c1ce3e0449bab5490b690721cbf P db0742ee3860468590eefef6ae6074fd -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.047638 7090 consensus_peers.cc:485] T 030335599b754c81810fe0d139ef51f1 P db0742ee3860468590eefef6ae6074fd -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:40.055176 4538 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:40.092307 12503 raft_consensus.cc:482] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.092484 12503 raft_consensus.cc:504] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } }
I20240501 20:10:40.093060 12503 leader_election.cc:288] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
W20240501 20:10:40.094241 4564 leader_election.cc:334] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
W20240501 20:10:40.132436 4078 catalog_manager.cc:4573] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet aecca0e22b084a8b95c98ef7e7b90b29: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
W20240501 20:10:40.142284 7222 consensus_peers.cc:485] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 -> Peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Couldn't send request to peer 10b47782558c41ee91f5e164a372049a. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:40.157490 12205 raft_consensus.cc:482] T bc8e3edd983e4be9a9726e73f04f08e3 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.157724 12205 raft_consensus.cc:504] T bc8e3edd983e4be9a9726e73f04f08e3 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:40.158382 12205 leader_election.cc:288] T bc8e3edd983e4be9a9726e73f04f08e3 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
W20240501 20:10:40.160007 7355 leader_election.cc:334] T bc8e3edd983e4be9a9726e73f04f08e3 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111)
W20240501 20:10:40.160220 7355 leader_election.cc:334] T bc8e3edd983e4be9a9726e73f04f08e3 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
I20240501 20:10:40.160310 7355 leader_election.cc:302] T bc8e3edd983e4be9a9726e73f04f08e3 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5; no voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530
I20240501 20:10:40.160547 12205 raft_consensus.cc:2726] T bc8e3edd983e4be9a9726e73f04f08e3 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:10:40.190412 7355 consensus_peers.cc:485] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:10:40.192979 7090 consensus_peers.cc:462] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:40.193171 12149 raft_consensus.cc:482] T a8eefa22b6e74b92b9fef036dea46e3b P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.193279 12149 raft_consensus.cc:504] T a8eefa22b6e74b92b9fef036dea46e3b P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:40.193795 12149 leader_election.cc:288] T a8eefa22b6e74b92b9fef036dea46e3b P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:40.194186 4631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8eefa22b6e74b92b9fef036dea46e3b" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
W20240501 20:10:40.195225 7090 leader_election.cc:334] T a8eefa22b6e74b92b9fef036dea46e3b P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
I20240501 20:10:40.195331 7090 leader_election.cc:302] T a8eefa22b6e74b92b9fef036dea46e3b P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db0742ee3860468590eefef6ae6074fd; no voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, dca57823003e4729b769ca12367d7530
I20240501 20:10:40.195519 12149 raft_consensus.cc:2726] T a8eefa22b6e74b92b9fef036dea46e3b P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:40.196344 12205 raft_consensus.cc:482] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.196528 12205 raft_consensus.cc:504] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:40.197167 12205 leader_election.cc:288] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:40.198262 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82260307259e4a9fb12231ff4abd79c5" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
W20240501 20:10:40.199182 7355 leader_election.cc:334] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
I20240501 20:10:40.199501 7353 leader_election.cc:302] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5; no voters: db0742ee3860468590eefef6ae6074fd, dca57823003e4729b769ca12367d7530
I20240501 20:10:40.199929 12205 raft_consensus.cc:2726] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:40.203271 12149 raft_consensus.cc:482] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.203380 12149 raft_consensus.cc:504] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:40.203866 12149 leader_election.cc:288] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:40.204236 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82260307259e4a9fb12231ff4abd79c5" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:40.204412 7419 raft_consensus.cc:2454] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:40.204795 7091 leader_election.cc:302] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:40.205219 12149 raft_consensus.cc:2781] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:10:40.205300 12149 raft_consensus.cc:482] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting leader election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.205391 12149 raft_consensus.cc:3037] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:40.207563 12149 raft_consensus.cc:504] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:40.208355 12149 leader_election.cc:288] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 election: Requested vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:40.208796 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82260307259e4a9fb12231ff4abd79c5" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:40.208999 7419 raft_consensus.cc:3037] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:10:40.209367 7090 leader_election.cc:334] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
W20240501 20:10:40.210234 7090 leader_election.cc:334] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
I20240501 20:10:40.211654 7419 raft_consensus.cc:2454] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 2.
I20240501 20:10:40.212038 7091 leader_election.cc:302] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters: dca57823003e4729b769ca12367d7530
I20240501 20:10:40.212276 12149 raft_consensus.cc:2781] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:40.212390 12149 raft_consensus.cc:686] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd [term 2 LEADER]: Becoming Leader. State: Replica: db0742ee3860468590eefef6ae6074fd, State: Running, Role: LEADER
I20240501 20:10:40.212622 12149 consensus_queue.cc:237] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:40.216634 4093 catalog_manager.cc:5484] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd reported cstate change: term changed from 1 to 2, leader changed from dca57823003e4729b769ca12367d7530 (127.29.219.65) to db0742ee3860468590eefef6ae6074fd (127.29.219.70). New cstate: current_term: 2 leader_uuid: "db0742ee3860468590eefef6ae6074fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:40.242982 12492 raft_consensus.cc:482] T d13178f10db0435891b97cebaed6dd5d P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.243165 12492 raft_consensus.cc:504] T d13178f10db0435891b97cebaed6dd5d P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:40.243793 12492 leader_election.cc:288] T d13178f10db0435891b97cebaed6dd5d P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
W20240501 20:10:40.244966 7222 leader_election.cc:334] T d13178f10db0435891b97cebaed6dd5d P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
W20240501 20:10:40.246505 7090 consensus_peers.cc:485] T 024c925714cb419aaa8870a053a6ca2f P db0742ee3860468590eefef6ae6074fd -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:10:40.309084 12149 raft_consensus.cc:482] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.309267 12149 raft_consensus.cc:504] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:40.309796 12149 leader_election.cc:288] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:40.310308 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ddbf0c6a4e4c1fb3c8d59288894a35" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
W20240501 20:10:40.311036 7090 leader_election.cc:334] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
I20240501 20:10:40.311147 7090 leader_election.cc:302] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db0742ee3860468590eefef6ae6074fd; no voters: b9957d4086bc4a76b18ed9265a5739d5, dca57823003e4729b769ca12367d7530
I20240501 20:10:40.311360 12149 raft_consensus.cc:2726] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:10:40.317828 7355 consensus_peers.cc:485] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:10:40.343225 12205 raft_consensus.cc:482] T 264d3fcd3fae4acd8d76306c0f32ed72 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.343463 12205 raft_consensus.cc:504] T 264d3fcd3fae4acd8d76306c0f32ed72 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:40.344157 12205 leader_election.cc:288] T 264d3fcd3fae4acd8d76306c0f32ed72 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
W20240501 20:10:40.346025 7355 leader_election.cc:334] T 264d3fcd3fae4acd8d76306c0f32ed72 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
I20240501 20:10:40.361634 12205 raft_consensus.cc:482] T 5f25c23e11dc48ee9d9496c1ae45fcaa P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.361694 12379 raft_consensus.cc:482] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.361858 12205 raft_consensus.cc:504] T 5f25c23e11dc48ee9d9496c1ae45fcaa P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:40.361876 12379 raft_consensus.cc:504] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:40.362453 12379 leader_election.cc:288] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:40.363047 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f25c23e11dc48ee9d9496c1ae45fcaa" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
W20240501 20:10:40.363839 7355 leader_election.cc:334] T 5f25c23e11dc48ee9d9496c1ae45fcaa P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
I20240501 20:10:40.363947 7355 leader_election.cc:302] T 5f25c23e11dc48ee9d9496c1ae45fcaa P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5; no voters: bceeae1c79964acabdae8ff771bf0c97, dca57823003e4729b769ca12367d7530
I20240501 20:10:40.364008 12205 leader_election.cc:288] T 5f25c23e11dc48ee9d9496c1ae45fcaa P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:40.364176 12379 raft_consensus.cc:2726] T 5f25c23e11dc48ee9d9496c1ae45fcaa P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:10:40.364259 7355 leader_election.cc:334] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
W20240501 20:10:40.364449 7355 leader_election.cc:334] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:40.364545 7355 leader_election.cc:302] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5; no voters: dca57823003e4729b769ca12367d7530, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:40.364771 12379 raft_consensus.cc:2726] T 4471aeb3f16f49d9bac12befd4cfd7e7 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:10:40.368175 12379 raft_consensus.cc:482] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.368345 12379 raft_consensus.cc:504] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:40.368988 12379 leader_election.cc:288] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
W20240501 20:10:40.370067 7355 leader_election.cc:334] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
I20240501 20:10:40.419488 12431 raft_consensus.cc:482] T e84a2462cb7e4e559fba887790a65a1f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.419648 12431 raft_consensus.cc:504] T e84a2462cb7e4e559fba887790a65a1f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:40.419807 11050 raft_consensus.cc:482] T f44675b6874a423398de361c7ccfd583 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.419987 11050 raft_consensus.cc:504] T f44675b6874a423398de361c7ccfd583 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:40.420194 12431 leader_election.cc:288] T e84a2462cb7e4e559fba887790a65a1f P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:40.420609 11050 leader_election.cc:288] T f44675b6874a423398de361c7ccfd583 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:40.425917 12379 raft_consensus.cc:482] T 53be6ced1df34676b1a34bc5cfe1ac87 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.426200 11050 raft_consensus.cc:482] T 9d634c1feb7449f686cf3af2a07181e0 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.426160 12379 raft_consensus.cc:504] T 53be6ced1df34676b1a34bc5cfe1ac87 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:40.426388 11050 raft_consensus.cc:504] T 9d634c1feb7449f686cf3af2a07181e0 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: false } }
I20240501 20:10:40.426976 11050 leader_election.cc:288] T 9d634c1feb7449f686cf3af2a07181e0 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
I20240501 20:10:40.427021 12379 leader_election.cc:288] T 53be6ced1df34676b1a34bc5cfe1ac87 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:40.431452 4498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "264d3fcd3fae4acd8d76306c0f32ed72" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:40.431695 4494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d13178f10db0435891b97cebaed6dd5d" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:40.431870 4496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84a2462cb7e4e559fba887790a65a1f" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:40.432018 4496 raft_consensus.cc:2454] T e84a2462cb7e4e559fba887790a65a1f P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:40.432252 4499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53be6ced1df34676b1a34bc5cfe1ac87" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:40.433178 4494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ae88161b514e4aa4b8d134a19de001" candidate_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3e974c03d3c546f9ab6c91e762a5426b" is_pre_election: true
I20240501 20:10:40.433343 4494 raft_consensus.cc:2454] T 00ae88161b514e4aa4b8d134a19de001 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d34e18a1d394d3f91c9f50d1c7bea4e in term 1.
I20240501 20:10:40.434360 12431 raft_consensus.cc:482] T 00ae88161b514e4aa4b8d134a19de001 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.434486 12431 raft_consensus.cc:504] T 00ae88161b514e4aa4b8d134a19de001 P 3e974c03d3c546f9ab6c91e762a5426b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } }
I20240501 20:10:40.435031 12431 leader_election.cc:288] T 00ae88161b514e4aa4b8d134a19de001 P 3e974c03d3c546f9ab6c91e762a5426b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:40.436278 7222 leader_election.cc:302] T d13178f10db0435891b97cebaed6dd5d P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97; no voters: 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530
I20240501 20:10:40.436669 12489 raft_consensus.cc:2726] T d13178f10db0435891b97cebaed6dd5d P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:40.439312 4564 leader_election.cc:302] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b; no voters: dca57823003e4729b769ca12367d7530
I20240501 20:10:40.439538 11999 raft_consensus.cc:2781] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:10:40.439606 11999 raft_consensus.cc:482] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting leader election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.439669 11999 raft_consensus.cc:3037] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:40.441836 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 4416
I20240501 20:10:40.443574 11999 raft_consensus.cc:504] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } }
I20240501 20:10:40.443904 7355 leader_election.cc:302] T 264d3fcd3fae4acd8d76306c0f32ed72 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5; no voters: 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530
I20240501 20:10:40.444190 12379 raft_consensus.cc:2726] T 264d3fcd3fae4acd8d76306c0f32ed72 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:40.444424 11999 leader_election.cc:288] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:40.444598 7355 leader_election.cc:302] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3e974c03d3c546f9ab6c91e762a5426b, b9957d4086bc4a76b18ed9265a5739d5; no voters: dca57823003e4729b769ca12367d7530
I20240501 20:10:40.444844 12379 raft_consensus.cc:2781] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:10:40.444926 12379 raft_consensus.cc:482] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.445008 12379 raft_consensus.cc:3037] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:10:40.445750 4564 leader_election.cc:334] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
I20240501 20:10:40.447301 12379 raft_consensus.cc:504] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:40.447883 12379 leader_election.cc:288] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 election: Requested vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
W20240501 20:10:40.449077 7355 leader_election.cc:334] T 53be6ced1df34676b1a34bc5cfe1ac87 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
I20240501 20:10:40.449185 7355 leader_election.cc:302] T 53be6ced1df34676b1a34bc5cfe1ac87 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5; no voters: 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530
I20240501 20:10:40.449440 12379 raft_consensus.cc:2726] T 53be6ced1df34676b1a34bc5cfe1ac87 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:10:40.450063 7355 leader_election.cc:334] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
I20240501 20:10:40.454715 12149 raft_consensus.cc:482] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:40.454900 12149 raft_consensus.cc:504] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { replace: true } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:40.455578 12149 leader_election.cc:288] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
W20240501 20:10:40.457640 7090 leader_election.cc:334] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111)
W20240501 20:10:40.457813 7090 leader_election.cc:334] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:40.457904 7090 leader_election.cc:302] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: db0742ee3860468590eefef6ae6074fd; no voters: 10b47782558c41ee91f5e164a372049a, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:40.458181 12149 raft_consensus.cc:2726] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:10:40.458166 7090 leader_election.cc:334] T cd21a0faa3bb487cb80f0951b12b3d67 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
I20240501 20:10:40.492379 12149 raft_consensus.cc:482] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.492586 12149 raft_consensus.cc:504] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:40.493260 12149 leader_election.cc:288] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:40.493896 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d690199665174c7d9bdbd54d2af4ac70" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:40.494486 12379 raft_consensus.cc:482] T 1d3bb8370e7c41dc9cf7f1c835ac275b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dca57823003e4729b769ca12367d7530)
W20240501 20:10:40.494588 7090 leader_election.cc:334] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
I20240501 20:10:40.494719 7090 leader_election.cc:302] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db0742ee3860468590eefef6ae6074fd; no voters: b9957d4086bc4a76b18ed9265a5739d5, dca57823003e4729b769ca12367d7530
I20240501 20:10:40.494728 12379 raft_consensus.cc:504] T 1d3bb8370e7c41dc9cf7f1c835ac275b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:40.494940 12149 raft_consensus.cc:2726] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:40.495682 12379 leader_election.cc:288] T 1d3bb8370e7c41dc9cf7f1c835ac275b P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
W20240501 20:10:40.496976 7355 leader_election.cc:334] T 1d3bb8370e7c41dc9cf7f1c835ac275b P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111)
W20240501 20:10:40.497186 7355 leader_election.cc:334] T 1d3bb8370e7c41dc9cf7f1c835ac275b P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
I20240501 20:10:40.497270 7355 leader_election.cc:302] T 1d3bb8370e7c41dc9cf7f1c835ac275b P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5; no voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530
I20240501 20:10:40.497553 12379 raft_consensus.cc:2726] T 1d3bb8370e7c41dc9cf7f1c835ac275b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:10:40.502501 7354 connection.cc:535] server connection from 127.29.219.67:43167 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:40.503222 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 4547
W20240501 20:10:40.503312 7222 connection.cc:535] client connection to 127.29.219.67:34595 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:40.503808 4564 connection.cc:535] client connection to 127.29.219.67:34595 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:40.504981 7222 consensus_peers.cc:485] T 376ae6034f394a0fb0cab0d8a9241134 P bceeae1c79964acabdae8ff771bf0c97 -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.67:34595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.505105 7222 consensus_peers.cc:485] T dfaa3794205d482c90aa4f860fa99cf3 P bceeae1c79964acabdae8ff771bf0c97 -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.67:34595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.505890 4564 connection.cc:535] server connection from 127.29.219.67:55693 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:40.506891 4564 leader_election.cc:334] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Network error: Client connection negotiation failed: client connection to 127.29.219.67:34595: connect: Connection refused (error 111)
I20240501 20:10:40.506997 4564 leader_election.cc:302] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d34e18a1d394d3f91c9f50d1c7bea4e; no voters: 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530
W20240501 20:10:40.507033 7355 connection.cc:535] client connection to 127.29.219.67:34595 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:40.507266 12503 raft_consensus.cc:2726] T 00ae88161b514e4aa4b8d134a19de001 P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240501 20:10:40.507351 4564 consensus_peers.cc:485] T 1c6c89fa7e6f485e889a5197da4bd135 P 1d34e18a1d394d3f91c9f50d1c7bea4e -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.67:34595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.509603 7355 leader_election.cc:334] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Network error: Client connection negotiation failed: client connection to 127.29.219.67:34595: connect: Connection refused (error 111)
I20240501 20:10:40.509768 7355 leader_election.cc:302] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5; no voters: 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530
I20240501 20:10:40.510391 12379 raft_consensus.cc:2726] T e84a2462cb7e4e559fba887790a65a1f P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:10:40.511433 4671 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:10:40.512176 7355 consensus_peers.cc:485] T 9c77905da1b3453fa9a83366f7c51c37 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.67:34595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.531947 7090 consensus_peers.cc:485] T 5215e3f2040747f29aa1c9181a628096 P db0742ee3860468590eefef6ae6074fd -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.67:34595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:40.533488 12379 raft_consensus.cc:482] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.533658 12379 raft_consensus.cc:504] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:40.534310 12379 leader_election.cc:288] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
W20240501 20:10:40.535698 7355 leader_election.cc:334] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
W20240501 20:10:40.536212 7355 leader_election.cc:334] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111)
I20240501 20:10:40.536345 7355 leader_election.cc:302] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5; no voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530
I20240501 20:10:40.536684 12379 raft_consensus.cc:2726] T 34cb2f3c90cc4158a61f426c0e5cb86b P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:40.553584 12149 raft_consensus.cc:482] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.553762 12149 raft_consensus.cc:504] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:40.554394 12149 leader_election.cc:288] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
W20240501 20:10:40.555984 7090 leader_election.cc:334] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
W20240501 20:10:40.556164 7090 leader_election.cc:334] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Network error: Client connection negotiation failed: client connection to 127.29.219.67:34595: connect: Connection refused (error 111)
I20240501 20:10:40.556264 7090 leader_election.cc:302] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db0742ee3860468590eefef6ae6074fd; no voters: 3e974c03d3c546f9ab6c91e762a5426b, dca57823003e4729b769ca12367d7530
I20240501 20:10:40.556587 12504 raft_consensus.cc:2726] T 56cb0964dabd41079d4428787efc9a9d P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:40.564365 12379 raft_consensus.cc:482] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.564555 12379 raft_consensus.cc:504] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:40.565376 12379 leader_election.cc:288] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:40.565821 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d690199665174c7d9bdbd54d2af4ac70" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:40.566021 7153 raft_consensus.cc:2454] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:40.566433 7353 leader_election.cc:302] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters:
W20240501 20:10:40.566663 7355 leader_election.cc:334] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
I20240501 20:10:40.566730 12379 raft_consensus.cc:2781] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:10:40.566891 12379 raft_consensus.cc:482] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.566982 12379 raft_consensus.cc:3037] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:40.568810 12379 raft_consensus.cc:504] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:40.569298 12379 leader_election.cc:288] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 election: Requested vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
I20240501 20:10:40.569852 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d690199665174c7d9bdbd54d2af4ac70" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:40.570032 7153 raft_consensus.cc:3037] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:10:40.570669 7355 leader_election.cc:334] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
I20240501 20:10:40.572265 7153 raft_consensus.cc:2454] T d690199665174c7d9bdbd54d2af4ac70 P db0742ee3860468590eefef6ae6074fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 2.
I20240501 20:10:40.572631 7353 leader_election.cc:302] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters: dca57823003e4729b769ca12367d7530
I20240501 20:10:40.573040 12205 raft_consensus.cc:2781] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:40.573221 12205 raft_consensus.cc:686] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 LEADER]: Becoming Leader. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Running, Role: LEADER
I20240501 20:10:40.573540 12205 consensus_queue.cc:237] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:40.577467 4093 catalog_manager.cc:5484] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: term changed from 1 to 2, leader changed from dca57823003e4729b769ca12367d7530 (127.29.219.65) to b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72). New cstate: current_term: 2 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:40.609584 12504 raft_consensus.cc:482] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10b47782558c41ee91f5e164a372049a)
I20240501 20:10:40.609751 12504 raft_consensus.cc:504] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:40.610327 12504 leader_election.cc:288] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
W20240501 20:10:40.612026 7090 leader_election.cc:334] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
W20240501 20:10:40.612198 7090 leader_election.cc:334] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111)
I20240501 20:10:40.612272 7090 leader_election.cc:302] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db0742ee3860468590eefef6ae6074fd; no voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530
I20240501 20:10:40.612470 12504 raft_consensus.cc:2726] T bb5700eb47e044c182c33183ebb27908 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:40.613423 12205 raft_consensus.cc:482] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.613579 12205 raft_consensus.cc:504] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:40.614245 12205 leader_election.cc:288] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:40.614646 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ddbf0c6a4e4c1fb3c8d59288894a35" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:40.614838 7153 raft_consensus.cc:2454] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 1.
I20240501 20:10:40.615411 7353 leader_election.cc:302] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:40.615664 12205 raft_consensus.cc:2781] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:10:40.615933 12205 raft_consensus.cc:482] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:40.616092 12205 raft_consensus.cc:3037] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:10:40.616379 7355 consensus_peers.cc:485] T c5b4fbeda51f40b8ac54baf5fb241710 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.67:34595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.617033 7355 leader_election.cc:334] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
I20240501 20:10:40.618566 12205 raft_consensus.cc:504] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:40.619143 12205 leader_election.cc:288] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 election: Requested vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:40.619534 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72ddbf0c6a4e4c1fb3c8d59288894a35" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd"
I20240501 20:10:40.619706 7153 raft_consensus.cc:3037] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:40.621579 7153 raft_consensus.cc:2454] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P db0742ee3860468590eefef6ae6074fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9957d4086bc4a76b18ed9265a5739d5 in term 2.
I20240501 20:10:40.621928 7353 leader_election.cc:302] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:40.622938 12205 raft_consensus.cc:2781] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:40.623354 12205 raft_consensus.cc:686] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 LEADER]: Becoming Leader. State: Replica: b9957d4086bc4a76b18ed9265a5739d5, State: Running, Role: LEADER
W20240501 20:10:40.623744 7355 leader_election.cc:334] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
I20240501 20:10:40.623744 12205 consensus_queue.cc:237] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } }
I20240501 20:10:40.629606 4093 catalog_manager.cc:5484] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5 reported cstate change: term changed from 1 to 2, leader changed from dca57823003e4729b769ca12367d7530 (127.29.219.65) to b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72). New cstate: current_term: 2 leader_uuid: "b9957d4086bc4a76b18ed9265a5739d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } }
W20240501 20:10:40.631585 7090 consensus_peers.cc:485] T 35c843bebb484f6cb655c146a02261ba P db0742ee3860468590eefef6ae6074fd -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.67:34595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:40.634776 12205 raft_consensus.cc:482] T 91a2bf5ddc03431b963ab5da831fd340 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10b47782558c41ee91f5e164a372049a)
I20240501 20:10:40.635031 12205 raft_consensus.cc:504] T 91a2bf5ddc03431b963ab5da831fd340 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:40.636211 12205 leader_election.cc:288] T 91a2bf5ddc03431b963ab5da831fd340 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
W20240501 20:10:40.637650 7355 leader_election.cc:334] T 91a2bf5ddc03431b963ab5da831fd340 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111)
W20240501 20:10:40.637815 7355 leader_election.cc:334] T 91a2bf5ddc03431b963ab5da831fd340 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
I20240501 20:10:40.637943 7355 leader_election.cc:302] T 91a2bf5ddc03431b963ab5da831fd340 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5; no voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530
I20240501 20:10:40.638195 12205 raft_consensus.cc:2726] T 91a2bf5ddc03431b963ab5da831fd340 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:10:40.652729 7090 consensus_peers.cc:485] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.656839 7090 consensus_peers.cc:485] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.67:34595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.677229 7355 consensus_peers.cc:485] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.67:34595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.714287 7090 consensus_peers.cc:462] T 67703d84deb9417bb21bde6365deda2f P db0742ee3860468590eefef6ae6074fd -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
W20240501 20:10:40.719065 7222 consensus_peers.cc:485] T 185ee8ce5e7f4617b0526b5f3fd64a29 P bceeae1c79964acabdae8ff771bf0c97 -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.67:34595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:40.747246 12205 raft_consensus.cc:482] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 10b47782558c41ee91f5e164a372049a)
I20240501 20:10:40.747296 12501 raft_consensus.cc:482] T c31148e9e4dd42c3a11787d9384c79c2 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10b47782558c41ee91f5e164a372049a)
I20240501 20:10:40.747550 12205 raft_consensus.cc:504] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:40.747596 12501 raft_consensus.cc:504] T c31148e9e4dd42c3a11787d9384c79c2 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:40.748268 12205 leader_election.cc:288] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:40.748461 12501 leader_election.cc:288] T c31148e9e4dd42c3a11787d9384c79c2 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
I20240501 20:10:40.749076 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffaf0a0105ea4e1f9ed9d90c221ebc8a" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
W20240501 20:10:40.750159 7355 leader_election.cc:334] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111)
I20240501 20:10:40.750260 7355 leader_election.cc:302] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5; no voters: 10b47782558c41ee91f5e164a372049a, bceeae1c79964acabdae8ff771bf0c97
I20240501 20:10:40.750519 12501 raft_consensus.cc:2726] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:10:40.750541 7355 leader_election.cc:334] T c31148e9e4dd42c3a11787d9384c79c2 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111)
I20240501 20:10:40.756036 7419 raft_consensus.cc:1261] T 82260307259e4a9fb12231ff4abd79c5 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Refusing update from remote peer db0742ee3860468590eefef6ae6074fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 20:10:40.756587 12504 consensus_queue.cc:1035] T 82260307259e4a9fb12231ff4abd79c5 P db0742ee3860468590eefef6ae6074fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 20:10:40.782440 7222 consensus_peers.cc:485] T aecca0e22b084a8b95c98ef7e7b90b29 P bceeae1c79964acabdae8ff771bf0c97 -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.67:34595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:40.789970 12517 raft_consensus.cc:482] T 0bf6bcf89d444b339612d731af604d0a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10b47782558c41ee91f5e164a372049a)
I20240501 20:10:40.790140 12517 raft_consensus.cc:504] T 0bf6bcf89d444b339612d731af604d0a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:40.790796 12517 leader_election.cc:288] T 0bf6bcf89d444b339612d731af604d0a P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
W20240501 20:10:40.792204 7222 leader_election.cc:334] T 0bf6bcf89d444b339612d731af604d0a P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111)
W20240501 20:10:40.792456 7222 leader_election.cc:334] T 0bf6bcf89d444b339612d731af604d0a P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
I20240501 20:10:40.792533 7222 leader_election.cc:302] T 0bf6bcf89d444b339612d731af604d0a P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97; no voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530
I20240501 20:10:40.792742 12517 raft_consensus.cc:2726] T 0bf6bcf89d444b339612d731af604d0a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:40.809315 12517 raft_consensus.cc:482] T ce89f9b962264db7a91685556f1354a8 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10b47782558c41ee91f5e164a372049a)
I20240501 20:10:40.809485 12517 raft_consensus.cc:504] T ce89f9b962264db7a91685556f1354a8 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:40.810271 12517 leader_election.cc:288] T ce89f9b962264db7a91685556f1354a8 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
W20240501 20:10:40.812775 7222 leader_election.cc:334] T ce89f9b962264db7a91685556f1354a8 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111)
W20240501 20:10:40.812925 7222 leader_election.cc:334] T ce89f9b962264db7a91685556f1354a8 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:40.812989 7222 leader_election.cc:302] T ce89f9b962264db7a91685556f1354a8 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97; no voters: 10b47782558c41ee91f5e164a372049a, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:40.813199 12517 raft_consensus.cc:2726] T ce89f9b962264db7a91685556f1354a8 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:10:40.815712 7222 consensus_peers.cc:485] T 965b65ec896648698fbf3e1266d8e678 P bceeae1c79964acabdae8ff771bf0c97 -> Peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Couldn't send request to peer 3e974c03d3c546f9ab6c91e762a5426b. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.67:34595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:40.817265 12504 raft_consensus.cc:482] T 6e3d6a53365f4eceac1d99d4cd51153b P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10b47782558c41ee91f5e164a372049a)
I20240501 20:10:40.817405 12504 raft_consensus.cc:504] T 6e3d6a53365f4eceac1d99d4cd51153b P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:40.817910 12504 leader_election.cc:288] T 6e3d6a53365f4eceac1d99d4cd51153b P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:40.818428 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3d6a53365f4eceac1d99d4cd51153b" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
W20240501 20:10:40.819377 7090 leader_election.cc:334] T 6e3d6a53365f4eceac1d99d4cd51153b P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111)
I20240501 20:10:40.819484 7090 leader_election.cc:302] T 6e3d6a53365f4eceac1d99d4cd51153b P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db0742ee3860468590eefef6ae6074fd; no voters: 10b47782558c41ee91f5e164a372049a, bceeae1c79964acabdae8ff771bf0c97
I20240501 20:10:40.819852 12504 raft_consensus.cc:2726] T 6e3d6a53365f4eceac1d99d4cd51153b P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:40.830186 12504 raft_consensus.cc:482] T 3a982d9467a14f88a54a240266be728d P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10b47782558c41ee91f5e164a372049a)
I20240501 20:10:40.830368 12504 raft_consensus.cc:504] T 3a982d9467a14f88a54a240266be728d P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:40.831007 12504 leader_election.cc:288] T 3a982d9467a14f88a54a240266be728d P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
W20240501 20:10:40.832278 7090 leader_election.cc:334] T 3a982d9467a14f88a54a240266be728d P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
W20240501 20:10:40.832443 7090 leader_election.cc:334] T 3a982d9467a14f88a54a240266be728d P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111)
I20240501 20:10:40.832525 7090 leader_election.cc:302] T 3a982d9467a14f88a54a240266be728d P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db0742ee3860468590eefef6ae6074fd; no voters: 10b47782558c41ee91f5e164a372049a, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:40.832726 12504 raft_consensus.cc:2726] T 3a982d9467a14f88a54a240266be728d P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:40.853642 12517 raft_consensus.cc:482] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 10b47782558c41ee91f5e164a372049a)
I20240501 20:10:40.853878 12517 raft_consensus.cc:504] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:40.855574 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffaf0a0105ea4e1f9ed9d90c221ebc8a" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:40.855868 7419 raft_consensus.cc:2454] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 2.
I20240501 20:10:40.856365 7223 leader_election.cc:302] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:40.856717 11935 raft_consensus.cc:2781] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:10:40.856820 11935 raft_consensus.cc:482] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 10b47782558c41ee91f5e164a372049a)
I20240501 20:10:40.856913 11935 raft_consensus.cc:3037] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:10:40.863134 12517 leader_election.cc:288] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:40.864176 11935 raft_consensus.cc:504] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
W20240501 20:10:40.864655 7222 leader_election.cc:334] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111)
I20240501 20:10:40.864849 11935 leader_election.cc:288] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 3 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:40.865233 11999 raft_consensus.cc:482] T bec3cae7dc9241fd8eae761967932f6e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10b47782558c41ee91f5e164a372049a)
I20240501 20:10:40.865378 11999 raft_consensus.cc:504] T bec3cae7dc9241fd8eae761967932f6e P 1d34e18a1d394d3f91c9f50d1c7bea4e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:40.865598 4627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c31148e9e4dd42c3a11787d9384c79c2" candidate_uuid: "b9957d4086bc4a76b18ed9265a5739d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" is_pre_election: true
I20240501 20:10:40.866078 11999 leader_election.cc:288] T bec3cae7dc9241fd8eae761967932f6e P 1d34e18a1d394d3f91c9f50d1c7bea4e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817)
I20240501 20:10:40.866274 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 4547
W20240501 20:10:40.867830 7222 leader_election.cc:334] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111)
I20240501 20:10:40.868608 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffaf0a0105ea4e1f9ed9d90c221ebc8a" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:40.868860 7419 raft_consensus.cc:3037] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:10:40.871196 7419 raft_consensus.cc:2454] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P b9957d4086bc4a76b18ed9265a5739d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 3.
I20240501 20:10:40.871614 7223 leader_election.cc:302] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, bceeae1c79964acabdae8ff771bf0c97; no voters: 10b47782558c41ee91f5e164a372049a
I20240501 20:10:40.871870 11935 raft_consensus.cc:2781] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:10:40.872011 11935 raft_consensus.cc:686] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [term 3 LEADER]: Becoming Leader. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Running, Role: LEADER
I20240501 20:10:40.872294 11935 consensus_queue.cc:237] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:40.875941 4088 catalog_manager.cc:5484] T ffaf0a0105ea4e1f9ed9d90c221ebc8a P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: term changed from 2 to 3, leader changed from 10b47782558c41ee91f5e164a372049a (127.29.219.66) to bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71). New cstate: current_term: 3 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:10:40.876879 12501 raft_consensus.cc:482] T bec3cae7dc9241fd8eae761967932f6e P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10b47782558c41ee91f5e164a372049a)
I20240501 20:10:40.877054 12501 raft_consensus.cc:504] T bec3cae7dc9241fd8eae761967932f6e P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { replace: true } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: NON_VOTER last_known_addr { host: "127.29.219.69" port: 46237 } attrs { promote: true } }
I20240501 20:10:40.877763 12501 leader_election.cc:288] T bec3cae7dc9241fd8eae761967932f6e P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
W20240501 20:10:40.879292 7355 leader_election.cc:334] T bec3cae7dc9241fd8eae761967932f6e P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111)
I20240501 20:10:40.879554 11935 raft_consensus.cc:482] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10b47782558c41ee91f5e164a372049a)
I20240501 20:10:40.879678 11935 raft_consensus.cc:504] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:40.880182 11935 leader_election.cc:288] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
W20240501 20:10:40.881482 7222 leader_election.cc:334] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111)
I20240501 20:10:40.882333 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66514deec5744da832bc030a582c46a" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:40.882862 7220 leader_election.cc:302] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97; no voters: 10b47782558c41ee91f5e164a372049a, db0742ee3860468590eefef6ae6074fd
I20240501 20:10:40.883179 11935 raft_consensus.cc:2726] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:10:40.922886 7089 connection.cc:535] server connection from 127.29.219.68:44315 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:40.922909 7356 connection.cc:535] client connection to 127.29.219.68:45205 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:40.923352 7223 connection.cc:535] server connection from 127.29.219.68:41363 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:40.924211 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 7071
W20240501 20:10:40.924288 7223 consensus_peers.cc:485] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97 -> Peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Couldn't send request to peer 1d34e18a1d394d3f91c9f50d1c7bea4e. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.68:45205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.924382 7223 consensus_peers.cc:485] T 46df0faa681b469abe9e383b181a6053 P bceeae1c79964acabdae8ff771bf0c97 -> Peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Couldn't send request to peer 1d34e18a1d394d3f91c9f50d1c7bea4e. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.68:45205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.924602 7091 consensus_peers.cc:485] T b2619f3bed304d3ab864f56d5b655574 P db0742ee3860468590eefef6ae6074fd -> Peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Couldn't send request to peer 1d34e18a1d394d3f91c9f50d1c7bea4e. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.68:45205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.924701 7091 consensus_peers.cc:485] T bb85d7825c564339bd554deeded45cec P db0742ee3860468590eefef6ae6074fd -> Peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Couldn't send request to peer 1d34e18a1d394d3f91c9f50d1c7bea4e. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.68:45205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.925182 7356 consensus_peers.cc:485] T 4f99f5d9d4bf44c39188c8b5d5362b98 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Couldn't send request to peer 1d34e18a1d394d3f91c9f50d1c7bea4e. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.68:45205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.925340 7356 leader_election.cc:334] T c31148e9e4dd42c3a11787d9384c79c2 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Network error: Client connection negotiation failed: client connection to 127.29.219.68:45205: connect: Connection refused (error 111)
I20240501 20:10:40.925494 7356 leader_election.cc:302] T c31148e9e4dd42c3a11787d9384c79c2 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5; no voters: 10b47782558c41ee91f5e164a372049a, 1d34e18a1d394d3f91c9f50d1c7bea4e
W20240501 20:10:40.925539 7091 consensus_peers.cc:485] T 9709a8bb107742b79078f91da9fab8f6 P db0742ee3860468590eefef6ae6074fd -> Peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Couldn't send request to peer 1d34e18a1d394d3f91c9f50d1c7bea4e. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.68:45205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.925612 7091 consensus_peers.cc:485] T c147c98b674e499992f0b9f556ac6815 P db0742ee3860468590eefef6ae6074fd -> Peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Couldn't send request to peer 1d34e18a1d394d3f91c9f50d1c7bea4e. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.68:45205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.925695 7091 consensus_peers.cc:485] T 4a474e1f29bf4f059b22b2ec01f15579 P db0742ee3860468590eefef6ae6074fd -> Peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Couldn't send request to peer 1d34e18a1d394d3f91c9f50d1c7bea4e. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.68:45205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:40.925859 12539 raft_consensus.cc:2726] T c31148e9e4dd42c3a11787d9384c79c2 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:10:40.925877 7356 consensus_peers.cc:485] T 9e31f8d1809244efae05ab9254c163ce P b9957d4086bc4a76b18ed9265a5739d5 -> Peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Couldn't send request to peer 1d34e18a1d394d3f91c9f50d1c7bea4e. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.68:45205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.926052 7356 leader_election.cc:334] T bec3cae7dc9241fd8eae761967932f6e P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Network error: Client connection negotiation failed: client connection to 127.29.219.68:45205: connect: Connection refused (error 111)
I20240501 20:10:40.926160 7356 leader_election.cc:302] T bec3cae7dc9241fd8eae761967932f6e P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5; no voters: 10b47782558c41ee91f5e164a372049a, 1d34e18a1d394d3f91c9f50d1c7bea4e
I20240501 20:10:40.926375 12539 raft_consensus.cc:2726] T bec3cae7dc9241fd8eae761967932f6e P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:10:40.926458 7356 consensus_peers.cc:485] T 9e0ecc1043af4bc1ab31dfa80ea47bb4 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Couldn't send request to peer 1d34e18a1d394d3f91c9f50d1c7bea4e. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.68:45205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.926571 7356 consensus_peers.cc:485] T 537b5ca846b5485ca1d98351d9038430 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Couldn't send request to peer 1d34e18a1d394d3f91c9f50d1c7bea4e. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.68:45205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:40.930300 7195 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:10:40.947697 7356 consensus_peers.cc:485] T 129322eab07149d2a6fb7eef96100e7a P b9957d4086bc4a76b18ed9265a5739d5 -> Peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Couldn't send request to peer 1d34e18a1d394d3f91c9f50d1c7bea4e. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.68:45205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:40.955862 7355 consensus_peers.cc:485] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:10:40.963371 12379 raft_consensus.cc:482] T ddcb6f5090f143c2b672e5630230878f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10b47782558c41ee91f5e164a372049a)
I20240501 20:10:40.963527 12379 raft_consensus.cc:504] T ddcb6f5090f143c2b672e5630230878f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:40.964133 12379 leader_election.cc:288] T ddcb6f5090f143c2b672e5630230878f P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
W20240501 20:10:40.965319 7356 leader_election.cc:334] T ddcb6f5090f143c2b672e5630230878f P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Network error: Client connection negotiation failed: client connection to 127.29.219.68:45205: connect: Connection refused (error 111)
W20240501 20:10:40.965447 7355 leader_election.cc:334] T ddcb6f5090f143c2b672e5630230878f P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111)
I20240501 20:10:40.965554 7355 leader_election.cc:302] T ddcb6f5090f143c2b672e5630230878f P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5; no voters: 10b47782558c41ee91f5e164a372049a, 1d34e18a1d394d3f91c9f50d1c7bea4e
I20240501 20:10:40.965756 12379 raft_consensus.cc:2726] T ddcb6f5090f143c2b672e5630230878f P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:40.982069 12489 raft_consensus.cc:482] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10b47782558c41ee91f5e164a372049a)
I20240501 20:10:40.982314 12489 raft_consensus.cc:504] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:40.983193 12489 leader_election.cc:288] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
W20240501 20:10:40.984390 7222 leader_election.cc:334] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111)
I20240501 20:10:41.044970 12379 raft_consensus.cc:482] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10b47782558c41ee91f5e164a372049a)
I20240501 20:10:41.045171 12379 raft_consensus.cc:504] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } }
I20240501 20:10:41.045846 12379 leader_election.cc:288] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
W20240501 20:10:41.047251 7355 leader_election.cc:334] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111)
W20240501 20:10:41.047463 7355 leader_election.cc:334] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Network error: Client connection negotiation failed: client connection to 127.29.219.67:34595: connect: Connection refused (error 111)
I20240501 20:10:41.047585 7355 leader_election.cc:302] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5; no voters: 10b47782558c41ee91f5e164a372049a, 3e974c03d3c546f9ab6c91e762a5426b
I20240501 20:10:41.047829 12379 raft_consensus.cc:2726] T 44aa74088ba24c338820957983d83e04 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:41.048725 12489 raft_consensus.cc:482] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:41.048871 12489 raft_consensus.cc:504] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } attrs { promote: false } }
I20240501 20:10:41.049340 11935 raft_consensus.cc:482] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:41.049427 12489 leader_election.cc:288] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dca57823003e4729b769ca12367d7530 (127.29.219.65:42941), 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689)
I20240501 20:10:41.049453 11935 raft_consensus.cc:504] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:41.049949 11935 leader_election.cc:288] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:41.050604 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f25c23e11dc48ee9d9496c1ae45fcaa" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5" is_pre_election: true
I20240501 20:10:41.051179 7419 raft_consensus.cc:2454] T 5f25c23e11dc48ee9d9496c1ae45fcaa P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:41.051613 7223 leader_election.cc:302] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, bceeae1c79964acabdae8ff771bf0c97; no voters:
I20240501 20:10:41.051901 11935 raft_consensus.cc:2781] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:10:41.051968 11935 raft_consensus.cc:482] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dca57823003e4729b769ca12367d7530)
I20240501 20:10:41.052022 11935 raft_consensus.cc:3037] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:10:41.052186 7222 leader_election.cc:334] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
W20240501 20:10:41.052289 7222 leader_election.cc:334] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
W20240501 20:10:41.052497 7222 leader_election.cc:334] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111)
I20240501 20:10:41.052567 7222 leader_election.cc:302] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97; no voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530
I20240501 20:10:41.052786 12489 raft_consensus.cc:2726] T 6cb1ca3dbf614f94b39e8ddd28ab04fc P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:41.054507 11935 raft_consensus.cc:504] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:41.055110 11935 leader_election.cc:288] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 election: Requested vote from peers b9957d4086bc4a76b18ed9265a5739d5 (127.29.219.72:39817), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:41.055760 7419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f25c23e11dc48ee9d9496c1ae45fcaa" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b9957d4086bc4a76b18ed9265a5739d5"
I20240501 20:10:41.056152 7419 raft_consensus.cc:3037] T 5f25c23e11dc48ee9d9496c1ae45fcaa P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:10:41.058225 7222 leader_election.cc:334] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
I20240501 20:10:41.059334 7419 raft_consensus.cc:2454] T 5f25c23e11dc48ee9d9496c1ae45fcaa P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 2.
I20240501 20:10:41.059731 7223 leader_election.cc:302] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5, bceeae1c79964acabdae8ff771bf0c97; no voters: dca57823003e4729b769ca12367d7530
I20240501 20:10:41.060118 11935 raft_consensus.cc:2781] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:10:41.060225 11935 raft_consensus.cc:686] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97 [term 2 LEADER]: Becoming Leader. State: Replica: bceeae1c79964acabdae8ff771bf0c97, State: Running, Role: LEADER
I20240501 20:10:41.060436 11935 consensus_queue.cc:237] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:41.064525 4088 catalog_manager.cc:5484] T 5f25c23e11dc48ee9d9496c1ae45fcaa P bceeae1c79964acabdae8ff771bf0c97 reported cstate change: term changed from 1 to 2, leader changed from dca57823003e4729b769ca12367d7530 (127.29.219.65) to bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71). New cstate: current_term: 2 leader_uuid: "bceeae1c79964acabdae8ff771bf0c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } health_report { overall_health: HEALTHY } } }
W20240501 20:10:41.071302 7223 consensus_peers.cc:485] T e7f35eafb0be4f4eab73133917b22bbf P bceeae1c79964acabdae8ff771bf0c97 -> Peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Couldn't send request to peer 1d34e18a1d394d3f91c9f50d1c7bea4e. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.68:45205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:41.137514 12489 raft_consensus.cc:482] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10b47782558c41ee91f5e164a372049a)
I20240501 20:10:41.137725 12489 raft_consensus.cc:504] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:41.138309 12489 leader_election.cc:288] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
W20240501 20:10:41.139758 7222 leader_election.cc:334] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111)
W20240501 20:10:41.139916 7222 leader_election.cc:334] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111)
I20240501 20:10:41.139993 7222 leader_election.cc:302] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97; no voters: 10b47782558c41ee91f5e164a372049a, dca57823003e4729b769ca12367d7530
I20240501 20:10:41.140185 12489 raft_consensus.cc:2726] T f40b062e103d4e54a913fa0f4111f623 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:41.153093 11935 raft_consensus.cc:482] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3e974c03d3c546f9ab6c91e762a5426b)
I20240501 20:10:41.153254 11935 raft_consensus.cc:504] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } attrs { promote: false } }
I20240501 20:10:41.153770 11935 leader_election.cc:288] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205)
W20240501 20:10:41.155087 7223 leader_election.cc:334] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205): Network error: Client connection negotiation failed: client connection to 127.29.219.68:45205: connect: Connection refused (error 111)
W20240501 20:10:41.155261 7222 leader_election.cc:334] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Network error: Client connection negotiation failed: client connection to 127.29.219.67:34595: connect: Connection refused (error 111)
I20240501 20:10:41.155354 7222 leader_election.cc:302] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97; no voters: 1d34e18a1d394d3f91c9f50d1c7bea4e, 3e974c03d3c546f9ab6c91e762a5426b
I20240501 20:10:41.155540 11935 raft_consensus.cc:2726] T 786296114e8a47c68df103934ec055d5 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:41.181421 11935 raft_consensus.cc:482] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3e974c03d3c546f9ab6c91e762a5426b)
I20240501 20:10:41.181594 11935 raft_consensus.cc:504] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } attrs { promote: false } }
I20240501 20:10:41.182188 11935 leader_election.cc:288] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
W20240501 20:10:41.183562 7222 leader_election.cc:334] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
W20240501 20:10:41.183732 7222 leader_election.cc:334] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Network error: Client connection negotiation failed: client connection to 127.29.219.67:34595: connect: Connection refused (error 111)
I20240501 20:10:41.183807 7222 leader_election.cc:302] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97; no voters: 3e974c03d3c546f9ab6c91e762a5426b, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:41.183997 11935 raft_consensus.cc:2726] T 7afb8e3154a64be78b8041b137b3681a P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:10:41.198169 7355 consensus_peers.cc:485] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:41.200608 11935 raft_consensus.cc:482] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3e974c03d3c546f9ab6c91e762a5426b)
I20240501 20:10:41.200760 11935 raft_consensus.cc:504] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:41.201280 11935 leader_election.cc:288] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
W20240501 20:10:41.202395 7222 leader_election.cc:334] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Network error: Client connection negotiation failed: client connection to 127.29.219.67:34595: connect: Connection refused (error 111)
W20240501 20:10:41.223356 7355 consensus_peers.cc:485] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:41.225898 4078 catalog_manager.cc:4573] TS ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 99bd0e9f4cd7496785ad71480d7d3d61: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:41.236670 12504 raft_consensus.cc:482] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10b47782558c41ee91f5e164a372049a)
I20240501 20:10:41.236873 12504 raft_consensus.cc:504] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } attrs { promote: false } }
I20240501 20:10:41.237632 12504 leader_election.cc:288] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447)
I20240501 20:10:41.248248 7152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4f195a6dd14be981395538c2a33ad9" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:41.248392 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3d6a53365f4eceac1d99d4cd51153b" candidate_uuid: "bceeae1c79964acabdae8ff771bf0c97" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "db0742ee3860468590eefef6ae6074fd" is_pre_election: true
I20240501 20:10:41.248647 7153 raft_consensus.cc:2454] T 6e3d6a53365f4eceac1d99d4cd51153b P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bceeae1c79964acabdae8ff771bf0c97 in term 1.
I20240501 20:10:41.249652 7286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66514deec5744da832bc030a582c46a" candidate_uuid: "db0742ee3860468590eefef6ae6074fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "bceeae1c79964acabdae8ff771bf0c97" is_pre_election: true
I20240501 20:10:41.249940 7286 raft_consensus.cc:2454] T c66514deec5744da832bc030a582c46a P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db0742ee3860468590eefef6ae6074fd in term 1.
I20240501 20:10:41.250011 7220 leader_election.cc:302] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97; no voters: 3e974c03d3c546f9ab6c91e762a5426b, db0742ee3860468590eefef6ae6074fd
I20240501 20:10:41.250479 7220 leader_election.cc:302] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, db0742ee3860468590eefef6ae6074fd; no voters: 10b47782558c41ee91f5e164a372049a
I20240501 20:10:41.250783 12517 raft_consensus.cc:2726] T ca4f195a6dd14be981395538c2a33ad9 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 20:10:41.250982 12517 raft_consensus.cc:2781] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:10:41.251047 12517 raft_consensus.cc:482] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10b47782558c41ee91f5e164a372049a)
I20240501 20:10:41.251128 12517 raft_consensus.cc:3037] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:41.251647 7089 leader_election.cc:302] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97, db0742ee3860468590eefef6ae6074fd; no voters:
I20240501 20:10:41.252002 12504 raft_consensus.cc:2781] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 20:10:41.252095 12504 raft_consensus.cc:482] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting leader election (detected failure of leader 10b47782558c41ee91f5e164a372049a)
I20240501 20:10:41.252171 12504 raft_consensus.cc:3037] T c66514deec5744da832bc030a582c46a P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:10:41.254959 12149 raft_consensus.cc:482] T 5a7b2f01b5684df5800987591a390a17 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10b47782558c41ee91f5e164a372049a)
I20240501 20:10:41.255088 12149 raft_consensus.cc:504] T 5a7b2f01b5684df5800987591a390a17 P db0742ee3860468590eefef6ae6074fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:41.255717 12149 leader_election.cc:288] T 5a7b2f01b5684df5800987591a390a17 P db0742ee3860468590eefef6ae6074fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:41.257347 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 7071
I20240501 20:10:41.254593 12517 raft_consensus.cc:504] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } }
I20240501 20:10:41.259356 12517 leader_election.cc:288] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 election: Requested vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321)
W20240501 20:10:41.261245 7222 leader_election.cc:334] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689): Network error: Client connection negotiation failed: client connection to 127.29.219.66:40689: connect: Connection refused (error 111)
W20240501 20:10:41.295931 7355 consensus_peers.cc:485] T 12f13fda83114ec5b9dcd0fa90ddef7e P b9957d4086bc4a76b18ed9265a5739d5 -> Peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Couldn't send request to peer ef0a4ada548446b3a531cb39a3c4e38d. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:10:41.307240 7353 connection.cc:535] server connection from 127.29.219.70:45989 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:41.307335 7220 connection.cc:535] client connection to 127.29.219.70:36321 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:41.307471 7353 connection.cc:535] client connection to 127.29.219.70:36321 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:41.307776 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 7204
W20240501 20:10:41.308625 7220 leader_election.cc:334] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Network error: Client connection negotiation failed: client connection to 127.29.219.70:36321: connect: Connection refused (error 111)
W20240501 20:10:41.308640 7353 consensus_peers.cc:485] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Couldn't send request to peer db0742ee3860468590eefef6ae6074fd. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.70:36321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:41.308770 7220 leader_election.cc:302] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bceeae1c79964acabdae8ff771bf0c97; no voters: 10b47782558c41ee91f5e164a372049a, db0742ee3860468590eefef6ae6074fd
W20240501 20:10:41.308801 7353 consensus_peers.cc:485] T 72ddbf0c6a4e4c1fb3c8d59288894a35 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Couldn't send request to peer db0742ee3860468590eefef6ae6074fd. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.70:36321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:41.308961 7353 consensus_peers.cc:485] T d690199665174c7d9bdbd54d2af4ac70 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Couldn't send request to peer db0742ee3860468590eefef6ae6074fd. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.70:36321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:41.309139 11935 raft_consensus.cc:2726] T 6e3d6a53365f4eceac1d99d4cd51153b P bceeae1c79964acabdae8ff771bf0c97 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 20:10:41.313925 7328 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 20:10:41.415062 7353 consensus_peers.cc:485] T 422e49e6312c48a2ab779701c35ed10e P b9957d4086bc4a76b18ed9265a5739d5 -> Peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Couldn't send request to peer db0742ee3860468590eefef6ae6074fd. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.70:36321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:41.459360 12539 raft_consensus.cc:482] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3e974c03d3c546f9ab6c91e762a5426b)
I20240501 20:10:41.459560 12539 raft_consensus.cc:504] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "db0742ee3860468590eefef6ae6074fd" member_type: VOTER last_known_addr { host: "127.29.219.70" port: 36321 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:41.460229 12539 leader_election.cc:288] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
W20240501 20:10:41.461565 7355 leader_election.cc:334] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Network error: Client connection negotiation failed: client connection to 127.29.219.67:34595: connect: Connection refused (error 111)
W20240501 20:10:41.461766 7353 leader_election.cc:334] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Network error: Client connection negotiation failed: client connection to 127.29.219.70:36321: connect: Connection refused (error 111)
I20240501 20:10:41.461899 7353 leader_election.cc:302] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5; no voters: 3e974c03d3c546f9ab6c91e762a5426b, db0742ee3860468590eefef6ae6074fd
I20240501 20:10:41.462141 12539 raft_consensus.cc:2726] T 99bd0e9f4cd7496785ad71480d7d3d61 P b9957d4086bc4a76b18ed9265a5739d5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:10:41.539683 12539 raft_consensus.cc:482] T 09dcd419921941e9a19cfd9a20b89c70 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3e974c03d3c546f9ab6c91e762a5426b)
I20240501 20:10:41.539875 12539 raft_consensus.cc:504] T 09dcd419921941e9a19cfd9a20b89c70 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "ef0a4ada548446b3a531cb39a3c4e38d" member_type: VOTER last_known_addr { host: "127.29.219.69" port: 46237 } }
I20240501 20:10:41.540421 12539 leader_election.cc:288] T 09dcd419921941e9a19cfd9a20b89c70 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595), ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237)
W20240501 20:10:41.541790 7355 leader_election.cc:334] T 09dcd419921941e9a19cfd9a20b89c70 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595): Network error: Client connection negotiation failed: client connection to 127.29.219.67:34595: connect: Connection refused (error 111)
W20240501 20:10:41.541975 7355 leader_election.cc:334] T 09dcd419921941e9a19cfd9a20b89c70 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ef0a4ada548446b3a531cb39a3c4e38d (127.29.219.69:46237): Network error: Client connection negotiation failed: client connection to 127.29.219.69:46237: connect: Connection refused (error 111)
I20240501 20:10:41.542114 7355 leader_election.cc:302] T 09dcd419921941e9a19cfd9a20b89c70 P b9957d4086bc4a76b18ed9265a5739d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9957d4086bc4a76b18ed9265a5739d5; no voters: 3e974c03d3c546f9ab6c91e762a5426b, ef0a4ada548446b3a531cb39a3c4e38d
I20240501 20:10:41.542346 12539 raft_consensus.cc:2726] T 09dcd419921941e9a19cfd9a20b89c70 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 20:10:41.607391 7220 consensus_peers.cc:485] T b67250c5773f4c9d9254d0306158ccde P bceeae1c79964acabdae8ff771bf0c97 -> Peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Couldn't send request to peer db0742ee3860468590eefef6ae6074fd. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.70:36321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:41.607561 7220 consensus_peers.cc:485] T 1283fda7ece44e199d74712fd8bba9da P bceeae1c79964acabdae8ff771bf0c97 -> Peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Couldn't send request to peer db0742ee3860468590eefef6ae6074fd. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.70:36321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:41.607652 7220 consensus_peers.cc:485] T c1e7ac38dc9c44d29e2d8127904407eb P bceeae1c79964acabdae8ff771bf0c97 -> Peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Couldn't send request to peer db0742ee3860468590eefef6ae6074fd. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.70:36321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:41.607755 7220 consensus_peers.cc:485] T ce0b97a1badb4935a4553902e6feb20b P bceeae1c79964acabdae8ff771bf0c97 -> Peer db0742ee3860468590eefef6ae6074fd (127.29.219.70:36321): Couldn't send request to peer db0742ee3860468590eefef6ae6074fd. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.70:36321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:41.611975 11935 raft_consensus.cc:482] T 26ed0760e9074bd896e55e99756602bb P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 10b47782558c41ee91f5e164a372049a)
I20240501 20:10:41.612131 11935 raft_consensus.cc:504] T 26ed0760e9074bd896e55e99756602bb P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "10b47782558c41ee91f5e164a372049a" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 40689 } } peers { permanent_uuid: "dca57823003e4729b769ca12367d7530" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 42941 } attrs { promote: false } }
I20240501 20:10:41.612674 11935 leader_election.cc:288] T 26ed0760e9074bd896e55e99756602bb P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 10b47782558c41ee91f5e164a372049a (127.29.219.66:40689), dca57823003e4729b769ca12367d7530 (127.29.219.65:42941)
I20240501 20:10:41.613057 12489 raft_consensus.cc:482] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3e974c03d3c546f9ab6c91e762a5426b)
I20240501 20:10:41.613176 12489 raft_consensus.cc:504] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } }
I20240501 20:10:41.613651 12489 leader_election.cc:288] T 6ced3d988d474a8e8ee8f0a1fb4cc88e P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:41.614202 12517 raft_consensus.cc:482] T 4613582180754f729afc8d0b44356e97 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3e974c03d3c546f9ab6c91e762a5426b)
I20240501 20:10:41.614316 12517 raft_consensus.cc:504] T 4613582180754f729afc8d0b44356e97 P bceeae1c79964acabdae8ff771bf0c97 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "bceeae1c79964acabdae8ff771bf0c97" member_type: VOTER last_known_addr { host: "127.29.219.71" port: 39447 } attrs { promote: false } }
I20240501 20:10:41.614884 12517 leader_election.cc:288] T 4613582180754f729afc8d0b44356e97 P bceeae1c79964acabdae8ff771bf0c97 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1d34e18a1d394d3f91c9f50d1c7bea4e (127.29.219.68:45205), 3e974c03d3c546f9ab6c91e762a5426b (127.29.219.67:34595)
I20240501 20:10:41.615460 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 7204
W20240501 20:10:41.632802 7355 consensus_peers.cc:485] T accaff8513134f96b5502b5c5385d892 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer dca57823003e4729b769ca12367d7530 (127.29.219.65:42941): Couldn't send request to peer dca57823003e4729b769ca12367d7530. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.65:42941: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:10:41.662086 7353 connection.cc:535] server connection from 127.29.219.71:54061 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:41.662153 7354 connection.cc:535] client connection to 127.29.219.71:39447 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:10:41.662487 4078 connection.cc:535] server connection from 127.29.219.71:43721 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 20:10:41.662683 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 7337
W20240501 20:10:41.663300 7354 consensus_peers.cc:485] T cf05708a20a04c66875867f5453f7fb9 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Couldn't send request to peer bceeae1c79964acabdae8ff771bf0c97. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.71:39447: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:10:41.663431 7354 consensus_peers.cc:485] T 55da8e421b23413f96578a4af135b499 P b9957d4086bc4a76b18ed9265a5739d5 -> Peer bceeae1c79964acabdae8ff771bf0c97 (127.29.219.71:39447): Couldn't send request to peer bceeae1c79964acabdae8ff771bf0c97. Status: Network error: Client connection negotiation failed: client connection to 127.29.219.71:39447: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:10:41.668390 7460 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:41.946223 12501 raft_consensus.cc:482] T ef7c0a48f11945d3953f3461d06bc184 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3e974c03d3c546f9ab6c91e762a5426b)
I20240501 20:10:41.946506 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 7337
I20240501 20:10:41.946480 12501 raft_consensus.cc:504] T ef7c0a48f11945d3953f3461d06bc184 P b9957d4086bc4a76b18ed9265a5739d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3e974c03d3c546f9ab6c91e762a5426b" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 34595 } } peers { permanent_uuid: "1d34e18a1d394d3f91c9f50d1c7bea4e" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 45205 } } peers { permanent_uuid: "b9957d4086bc4a76b18ed9265a5739d5" member_type: VOTER last_known_addr { host: "127.29.219.72" port: 39817 } attrs { promote: false } }
I20240501 20:10:41.991689 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 4061
I20240501 20:10:41.997262 4123 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:42.279649 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 4061
2024-05-01T20:10:42Z chronyd exiting
I20240501 20:10:42.339185 30573 test_util.cc:175] -----------------------------------------------
I20240501 20:10:42.339289 30573 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714594112840080-30573-0
[ FAILED ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1, where GetParam() = 4-byte object <01-00 00-00> (82108 ms)
[----------] 1 test from DDLDuringRebalancingTest (82108 ms total)
[----------] 1 test from RebalancingDuringElectionStormTest
[ RUN ] RebalancingDuringElectionStormTest.RoundRobin/1
I20240501 20:10:42.341336 30573 test_util.cc:268] Using random seed: 348105143
I20240501 20:10:42.343221 30573 ts_itest-base.cc:112] Starting cluster with:
I20240501 20:10:42.343291 30573 ts_itest-base.cc:113] --------------
I20240501 20:10:42.343328 30573 ts_itest-base.cc:114] 8 tablet servers
I20240501 20:10:42.343381 30573 ts_itest-base.cc:115] 3 replicas per TS
I20240501 20:10:42.343418 30573 ts_itest-base.cc:116] --------------
2024-05-01T20:10:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T20:10:42Z Disabled control of system clock
I20240501 20:10:42.379782 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.219.126:34159
--webserver_interface=127.29.219.126
--webserver_port=0
--builtin_ntp_servers=127.29.219.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.219.126:34159
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 20:10:42.747807 12574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:42.748107 12574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:42.748227 12574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:42.755673 12574 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:42.755800 12574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 20:10:42.755877 12574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:42.755954 12574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 20:10:42.756024 12574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 20:10:42.765522 12574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.29.219.126:34159
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.219.126:34159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.29.219.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:42.766919 12574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:42.768764 12574 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:42.780170 12579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:42.780767 12580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:42.781641 12582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:42.782415 12574 server_base.cc:1034] running on GCE node
I20240501 20:10:42.783238 12574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:42.784696 12574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:42.785909 12574 hybrid_clock.cc:648] HybridClock initialized: now 1714594242785861 us; error 39 us; skew 500 ppm
I20240501 20:10:42.786260 12574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:42.788636 12574 webserver.cc:460] Webserver started at http://127.29.219.126:45005/ using document root <none> and password file <none>
I20240501 20:10:42.789382 12574 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:42.789497 12574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:42.789812 12574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:42.792579 12574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0361148318b0455795acb3989cd2c660"
format_stamp: "Formatted at 2024-05-01 20:10:42 on dist-test-slave-h2cj"
I20240501 20:10:42.793377 12574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0361148318b0455795acb3989cd2c660"
format_stamp: "Formatted at 2024-05-01 20:10:42 on dist-test-slave-h2cj"
I20240501 20:10:42.798660 12574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240501 20:10:42.802070 12588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:42.803395 12574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240501 20:10:42.803571 12574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0361148318b0455795acb3989cd2c660"
format_stamp: "Formatted at 2024-05-01 20:10:42 on dist-test-slave-h2cj"
I20240501 20:10:42.803764 12574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:42.827476 12574 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:10:42.828343 12574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:42.828642 12574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:42.846766 12574 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.126:34159
I20240501 20:10:42.846786 12639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.126:34159 every 8 connection(s)
I20240501 20:10:42.848096 12574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 20:10:42.851646 12640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:42.852519 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 12574
I20240501 20:10:42.852773 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 20:10:42.863742 12640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0361148318b0455795acb3989cd2c660: Bootstrap starting.
I20240501 20:10:42.867267 12640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0361148318b0455795acb3989cd2c660: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:42.868296 12640 log.cc:824] T 00000000000000000000000000000000 P 0361148318b0455795acb3989cd2c660: Log is configured to *not* fsync() on all Append() calls
I20240501 20:10:42.870810 12640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0361148318b0455795acb3989cd2c660: No bootstrap required, opened a new log
I20240501 20:10:42.877291 12640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0361148318b0455795acb3989cd2c660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0361148318b0455795acb3989cd2c660" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 34159 } }
I20240501 20:10:42.877741 12640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0361148318b0455795acb3989cd2c660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:42.877858 12640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0361148318b0455795acb3989cd2c660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0361148318b0455795acb3989cd2c660, State: Initialized, Role: FOLLOWER
I20240501 20:10:42.878530 12640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0361148318b0455795acb3989cd2c660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0361148318b0455795acb3989cd2c660" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 34159 } }
I20240501 20:10:42.878785 12640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0361148318b0455795acb3989cd2c660 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 20:10:42.878906 12640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0361148318b0455795acb3989cd2c660 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 20:10:42.879066 12640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0361148318b0455795acb3989cd2c660 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:42.881172 12640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0361148318b0455795acb3989cd2c660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0361148318b0455795acb3989cd2c660" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 34159 } }
I20240501 20:10:42.881693 12640 leader_election.cc:302] T 00000000000000000000000000000000 P 0361148318b0455795acb3989cd2c660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0361148318b0455795acb3989cd2c660; no voters:
I20240501 20:10:42.882331 12640 leader_election.cc:288] T 00000000000000000000000000000000 P 0361148318b0455795acb3989cd2c660 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 20:10:42.882494 12645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0361148318b0455795acb3989cd2c660 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:42.883226 12645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0361148318b0455795acb3989cd2c660 [term 1 LEADER]: Becoming Leader. State: Replica: 0361148318b0455795acb3989cd2c660, State: Running, Role: LEADER
I20240501 20:10:42.883797 12645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0361148318b0455795acb3989cd2c660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0361148318b0455795acb3989cd2c660" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 34159 } }
I20240501 20:10:42.884327 12640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0361148318b0455795acb3989cd2c660 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 20:10:42.886427 12647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0361148318b0455795acb3989cd2c660 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0361148318b0455795acb3989cd2c660. Latest consensus state: current_term: 1 leader_uuid: "0361148318b0455795acb3989cd2c660" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0361148318b0455795acb3989cd2c660" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 34159 } } }
I20240501 20:10:42.886842 12647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0361148318b0455795acb3989cd2c660 [sys.catalog]: This master's current role is: LEADER
I20240501 20:10:42.887441 12646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0361148318b0455795acb3989cd2c660 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0361148318b0455795acb3989cd2c660" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0361148318b0455795acb3989cd2c660" member_type: VOTER last_known_addr { host: "127.29.219.126" port: 34159 } } }
I20240501 20:10:42.887792 12646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0361148318b0455795acb3989cd2c660 [sys.catalog]: This master's current role is: LEADER
I20240501 20:10:42.889287 12650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 20:10:42.894423 12650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 20:10:42.903952 12650 catalog_manager.cc:1347] Generated new cluster ID: 481f702b206f48a5a7eacd61256b290f
I20240501 20:10:42.904210 12650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 20:10:42.929718 12650 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 20:10:42.931103 12650 catalog_manager.cc:1504] Loading token signing keys...
I20240501 20:10:42.944339 12650 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0361148318b0455795acb3989cd2c660: Generated new TSK 0
I20240501 20:10:42.945199 12650 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 20:10:42.950289 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.65:0
--local_ip_for_outbound_sockets=127.29.219.65
--webserver_interface=127.29.219.65
--webserver_port=0
--tserver_master_addrs=127.29.219.126:34159
--builtin_ntp_servers=127.29.219.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:10:43.282354 12664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:43.282701 12664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:43.282848 12664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:43.290525 12664 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:43.290721 12664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:43.290903 12664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.65
I20240501 20:10:43.300215 12664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.29.219.65
--webserver_port=0
--tserver_master_addrs=127.29.219.126:34159
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.65
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:43.301584 12664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:43.303385 12664 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:43.314944 12670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:43.315586 12669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:43.316458 12672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:43.316761 12664 server_base.cc:1034] running on GCE node
I20240501 20:10:43.317360 12664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:43.318300 12664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:43.319496 12664 hybrid_clock.cc:648] HybridClock initialized: now 1714594243319444 us; error 48 us; skew 500 ppm
I20240501 20:10:43.319804 12664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:43.325980 12664 webserver.cc:460] Webserver started at http://127.29.219.65:44877/ using document root <none> and password file <none>
I20240501 20:10:43.326804 12664 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:43.326922 12664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:43.327270 12664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:43.329954 12664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4e9d799e45f8407590cb3096f5625c62"
format_stamp: "Formatted at 2024-05-01 20:10:43 on dist-test-slave-h2cj"
I20240501 20:10:43.330770 12664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4e9d799e45f8407590cb3096f5625c62"
format_stamp: "Formatted at 2024-05-01 20:10:43 on dist-test-slave-h2cj"
I20240501 20:10:43.336019 12664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 20:10:43.339483 12678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:43.340808 12664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240501 20:10:43.340970 12664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4e9d799e45f8407590cb3096f5625c62"
format_stamp: "Formatted at 2024-05-01 20:10:43 on dist-test-slave-h2cj"
I20240501 20:10:43.341182 12664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:43.356325 12664 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:10:43.357179 12664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:43.357441 12664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:43.358392 12664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:43.359864 12664 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:43.359966 12664 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:43.360075 12664 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:43.360143 12664 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:43.391631 12664 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.65:38629
I20240501 20:10:43.391690 12790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.65:38629 every 8 connection(s)
I20240501 20:10:43.392939 12664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 20:10:43.403508 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 12664
I20240501 20:10:43.403720 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 20:10:43.406335 12791 heartbeater.cc:347] Connected to a master server at 127.29.219.126:34159
I20240501 20:10:43.406656 12791 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:43.407212 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.66:0
--local_ip_for_outbound_sockets=127.29.219.66
--webserver_interface=127.29.219.66
--webserver_port=0
--tserver_master_addrs=127.29.219.126:34159
--builtin_ntp_servers=127.29.219.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:10:43.407516 12791 heartbeater.cc:511] Master 127.29.219.126:34159 requested a full tablet report, sending...
I20240501 20:10:43.409989 12605 ts_manager.cc:194] Registered new tserver with Master: 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:10:43.412263 12605 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.65:50933
W20240501 20:10:43.739445 12795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:43.739781 12795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:43.739917 12795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:43.747730 12795 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:43.747890 12795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:43.748046 12795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.66
I20240501 20:10:43.757201 12795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.29.219.66
--webserver_port=0
--tserver_master_addrs=127.29.219.126:34159
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.66
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:43.758563 12795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:43.760329 12795 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:43.771909 12800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:43.772542 12803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:43.773262 12801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:43.774055 12795 server_base.cc:1034] running on GCE node
I20240501 20:10:43.774714 12795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:43.775810 12795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:43.777048 12795 hybrid_clock.cc:648] HybridClock initialized: now 1714594243776982 us; error 48 us; skew 500 ppm
I20240501 20:10:43.777400 12795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:43.779815 12795 webserver.cc:460] Webserver started at http://127.29.219.66:37565/ using document root <none> and password file <none>
I20240501 20:10:43.780611 12795 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:43.780721 12795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:43.781059 12795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:43.783783 12795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "77bf705599644326b6e71b5caf32fc7c"
format_stamp: "Formatted at 2024-05-01 20:10:43 on dist-test-slave-h2cj"
I20240501 20:10:43.784593 12795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "77bf705599644326b6e71b5caf32fc7c"
format_stamp: "Formatted at 2024-05-01 20:10:43 on dist-test-slave-h2cj"
I20240501 20:10:43.789970 12795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20240501 20:10:43.793545 12809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:43.795198 12795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:10:43.795387 12795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "77bf705599644326b6e71b5caf32fc7c"
format_stamp: "Formatted at 2024-05-01 20:10:43 on dist-test-slave-h2cj"
I20240501 20:10:43.795610 12795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:43.809441 12795 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:10:43.810309 12795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:43.810570 12795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:43.811582 12795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:43.813035 12795 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:43.813119 12795 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:43.813202 12795 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:43.813249 12795 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:43.844560 12795 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.66:32867
I20240501 20:10:43.844578 12921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.66:32867 every 8 connection(s)
I20240501 20:10:43.845965 12795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 20:10:43.851727 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 12795
I20240501 20:10:43.851922 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 20:10:43.855242 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.67:0
--local_ip_for_outbound_sockets=127.29.219.67
--webserver_interface=127.29.219.67
--webserver_port=0
--tserver_master_addrs=127.29.219.126:34159
--builtin_ntp_servers=127.29.219.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:10:43.860488 12922 heartbeater.cc:347] Connected to a master server at 127.29.219.126:34159
I20240501 20:10:43.860857 12922 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:43.861960 12922 heartbeater.cc:511] Master 127.29.219.126:34159 requested a full tablet report, sending...
I20240501 20:10:43.863585 12605 ts_manager.cc:194] Registered new tserver with Master: 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:10:43.864672 12605 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.66:60165
W20240501 20:10:44.195201 12925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:44.195492 12925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:44.195590 12925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:44.203258 12925 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:44.203383 12925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:44.203506 12925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.67
I20240501 20:10:44.212913 12925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.29.219.67
--webserver_port=0
--tserver_master_addrs=127.29.219.126:34159
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.67
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:44.214207 12925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:44.215936 12925 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:44.228358 12930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:44.228317 12931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:44.229084 12933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:44.229804 12925 server_base.cc:1034] running on GCE node
I20240501 20:10:44.230430 12925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:44.231460 12925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:44.232699 12925 hybrid_clock.cc:648] HybridClock initialized: now 1714594244232632 us; error 54 us; skew 500 ppm
I20240501 20:10:44.233107 12925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:44.235546 12925 webserver.cc:460] Webserver started at http://127.29.219.67:41667/ using document root <none> and password file <none>
I20240501 20:10:44.236341 12925 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:44.236454 12925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:44.236774 12925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:44.242799 12925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b4720929b44c4560bfdf08107e137362"
format_stamp: "Formatted at 2024-05-01 20:10:44 on dist-test-slave-h2cj"
I20240501 20:10:44.243570 12925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b4720929b44c4560bfdf08107e137362"
format_stamp: "Formatted at 2024-05-01 20:10:44 on dist-test-slave-h2cj"
I20240501 20:10:44.248903 12925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:10:44.252259 12939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:44.253533 12925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 20:10:44.253684 12925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b4720929b44c4560bfdf08107e137362"
format_stamp: "Formatted at 2024-05-01 20:10:44 on dist-test-slave-h2cj"
I20240501 20:10:44.253882 12925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:44.268857 12925 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:10:44.269677 12925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:44.269956 12925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:44.270924 12925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:44.272357 12925 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:44.272452 12925 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:44.272567 12925 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:44.272637 12925 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:44.304167 12925 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.67:42701
I20240501 20:10:44.304224 13051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.67:42701 every 8 connection(s)
I20240501 20:10:44.305418 12925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 20:10:44.311851 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 12925
I20240501 20:10:44.312011 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 20:10:44.315124 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.68:0
--local_ip_for_outbound_sockets=127.29.219.68
--webserver_interface=127.29.219.68
--webserver_port=0
--tserver_master_addrs=127.29.219.126:34159
--builtin_ntp_servers=127.29.219.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:10:44.318383 13052 heartbeater.cc:347] Connected to a master server at 127.29.219.126:34159
I20240501 20:10:44.318732 13052 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:44.319591 13052 heartbeater.cc:511] Master 127.29.219.126:34159 requested a full tablet report, sending...
I20240501 20:10:44.321048 12605 ts_manager.cc:194] Registered new tserver with Master: b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:10:44.321866 12605 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.67:57909
I20240501 20:10:44.415887 12791 heartbeater.cc:503] Master 127.29.219.126:34159 was elected leader, sending a full tablet report...
W20240501 20:10:44.651602 13055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:44.651877 13055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:44.651962 13055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:44.659359 13055 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:44.659466 13055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:44.659575 13055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.68
I20240501 20:10:44.668224 13055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.29.219.68
--webserver_port=0
--tserver_master_addrs=127.29.219.126:34159
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.68
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:44.669476 13055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:44.671144 13055 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:44.682551 13060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:44.682864 13063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:44.683518 13061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:44.683728 13055 server_base.cc:1034] running on GCE node
I20240501 20:10:44.684302 13055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:44.685225 13055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:44.686434 13055 hybrid_clock.cc:648] HybridClock initialized: now 1714594244686389 us; error 34 us; skew 500 ppm
I20240501 20:10:44.686784 13055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:44.688947 13055 webserver.cc:460] Webserver started at http://127.29.219.68:33553/ using document root <none> and password file <none>
I20240501 20:10:44.689663 13055 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:44.689774 13055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:44.690096 13055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:44.692701 13055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "bb1be87a7aee48a4b405b9d209c61231"
format_stamp: "Formatted at 2024-05-01 20:10:44 on dist-test-slave-h2cj"
I20240501 20:10:44.693465 13055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "bb1be87a7aee48a4b405b9d209c61231"
format_stamp: "Formatted at 2024-05-01 20:10:44 on dist-test-slave-h2cj"
I20240501 20:10:44.698706 13055 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240501 20:10:44.702237 13069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:44.703517 13055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:10:44.703666 13055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "bb1be87a7aee48a4b405b9d209c61231"
format_stamp: "Formatted at 2024-05-01 20:10:44 on dist-test-slave-h2cj"
I20240501 20:10:44.703862 13055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:44.731053 13055 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:10:44.731913 13055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:44.732178 13055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:44.733119 13055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:44.734609 13055 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:44.734735 13055 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:44.734856 13055 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:44.734925 13055 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:44.765935 13055 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.68:37951
I20240501 20:10:44.765992 13181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.68:37951 every 8 connection(s)
I20240501 20:10:44.767285 13055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 20:10:44.768918 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 13055
I20240501 20:10:44.769117 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 20:10:44.772351 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.69:0
--local_ip_for_outbound_sockets=127.29.219.69
--webserver_interface=127.29.219.69
--webserver_port=0
--tserver_master_addrs=127.29.219.126:34159
--builtin_ntp_servers=127.29.219.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:10:44.780149 13182 heartbeater.cc:347] Connected to a master server at 127.29.219.126:34159
I20240501 20:10:44.780432 13182 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:44.781235 13182 heartbeater.cc:511] Master 127.29.219.126:34159 requested a full tablet report, sending...
I20240501 20:10:44.782600 12605 ts_manager.cc:194] Registered new tserver with Master: bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:10:44.783388 12605 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.68:34701
I20240501 20:10:44.867342 12922 heartbeater.cc:503] Master 127.29.219.126:34159 was elected leader, sending a full tablet report...
W20240501 20:10:45.106204 13185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:45.106537 13185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:45.106742 13185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:45.114535 13185 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:45.114711 13185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:45.114895 13185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.69
I20240501 20:10:45.124317 13185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.29.219.69
--webserver_port=0
--tserver_master_addrs=127.29.219.126:34159
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.69
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:45.125696 13185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:45.127489 13185 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:45.139196 13191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:45.139930 13190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:45.140578 13193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:45.140980 13185 server_base.cc:1034] running on GCE node
I20240501 20:10:45.141605 13185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:45.142673 13185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:45.143898 13185 hybrid_clock.cc:648] HybridClock initialized: now 1714594245143852 us; error 30 us; skew 500 ppm
I20240501 20:10:45.144259 13185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:45.146538 13185 webserver.cc:460] Webserver started at http://127.29.219.69:40979/ using document root <none> and password file <none>
I20240501 20:10:45.147382 13185 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:45.147537 13185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:45.147956 13185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:45.150573 13185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "ad74bd6a74e840919906ee61ce3560a4"
format_stamp: "Formatted at 2024-05-01 20:10:45 on dist-test-slave-h2cj"
I20240501 20:10:45.151360 13185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "ad74bd6a74e840919906ee61ce3560a4"
format_stamp: "Formatted at 2024-05-01 20:10:45 on dist-test-slave-h2cj"
I20240501 20:10:45.156365 13185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240501 20:10:45.159771 13199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:45.161023 13185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:10:45.161155 13185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "ad74bd6a74e840919906ee61ce3560a4"
format_stamp: "Formatted at 2024-05-01 20:10:45 on dist-test-slave-h2cj"
I20240501 20:10:45.161324 13185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:45.191057 13185 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:10:45.191936 13185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:45.192181 13185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:45.193116 13185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:45.194711 13185 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:45.194805 13185 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:45.194892 13185 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:45.194947 13185 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:45.226141 13185 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.69:33377
I20240501 20:10:45.226200 13311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.69:33377 every 8 connection(s)
I20240501 20:10:45.227470 13185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:10:45.235440 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 13185
I20240501 20:10:45.235625 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 20:10:45.239219 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.70:0
--local_ip_for_outbound_sockets=127.29.219.70
--webserver_interface=127.29.219.70
--webserver_port=0
--tserver_master_addrs=127.29.219.126:34159
--builtin_ntp_servers=127.29.219.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:10:45.240033 13312 heartbeater.cc:347] Connected to a master server at 127.29.219.126:34159
I20240501 20:10:45.240315 13312 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:45.241186 13312 heartbeater.cc:511] Master 127.29.219.126:34159 requested a full tablet report, sending...
I20240501 20:10:45.247081 12605 ts_manager.cc:194] Registered new tserver with Master: ad74bd6a74e840919906ee61ce3560a4 (127.29.219.69:33377)
I20240501 20:10:45.247941 12605 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.69:59627
I20240501 20:10:45.324394 13052 heartbeater.cc:503] Master 127.29.219.126:34159 was elected leader, sending a full tablet report...
W20240501 20:10:45.574062 13315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:45.574373 13315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:45.574471 13315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:45.582182 13315 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:45.582325 13315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:45.582509 13315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.70
I20240501 20:10:45.592042 13315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.29.219.70
--webserver_port=0
--tserver_master_addrs=127.29.219.126:34159
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.70
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:45.593448 13315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:45.595285 13315 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:45.607563 13320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:45.608245 13323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:45.608902 13321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:45.609166 13315 server_base.cc:1034] running on GCE node
I20240501 20:10:45.609777 13315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:45.610738 13315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:45.612046 13315 hybrid_clock.cc:648] HybridClock initialized: now 1714594245611981 us; error 54 us; skew 500 ppm
I20240501 20:10:45.612385 13315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:45.614698 13315 webserver.cc:460] Webserver started at http://127.29.219.70:45481/ using document root <none> and password file <none>
I20240501 20:10:45.615464 13315 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:45.615588 13315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:45.615911 13315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:45.618489 13315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "6c07aec53e944308aa4630aee955ba38"
format_stamp: "Formatted at 2024-05-01 20:10:45 on dist-test-slave-h2cj"
I20240501 20:10:45.619257 13315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "6c07aec53e944308aa4630aee955ba38"
format_stamp: "Formatted at 2024-05-01 20:10:45 on dist-test-slave-h2cj"
I20240501 20:10:45.624631 13315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:10:45.628167 13329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:45.629698 13315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 20:10:45.629922 13315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6c07aec53e944308aa4630aee955ba38"
format_stamp: "Formatted at 2024-05-01 20:10:45 on dist-test-slave-h2cj"
I20240501 20:10:45.630141 13315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:45.649050 13315 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:10:45.649914 13315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:45.650177 13315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:45.651157 13315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:45.652634 13315 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:45.652720 13315 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:45.652832 13315 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:45.652899 13315 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:45.683570 13315 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.70:41485
I20240501 20:10:45.683637 13441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.70:41485 every 8 connection(s)
I20240501 20:10:45.684851 13315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:10:45.694330 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 13315
I20240501 20:10:45.694532 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 20:10:45.697211 13442 heartbeater.cc:347] Connected to a master server at 127.29.219.126:34159
I20240501 20:10:45.697554 13442 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:45.697889 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.71:0
--local_ip_for_outbound_sockets=127.29.219.71
--webserver_interface=127.29.219.71
--webserver_port=0
--tserver_master_addrs=127.29.219.126:34159
--builtin_ntp_servers=127.29.219.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:10:45.698427 13442 heartbeater.cc:511] Master 127.29.219.126:34159 requested a full tablet report, sending...
I20240501 20:10:45.699971 12605 ts_manager.cc:194] Registered new tserver with Master: 6c07aec53e944308aa4630aee955ba38 (127.29.219.70:41485)
I20240501 20:10:45.701090 12605 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.70:40239
I20240501 20:10:45.785703 13182 heartbeater.cc:503] Master 127.29.219.126:34159 was elected leader, sending a full tablet report...
W20240501 20:10:46.041462 13445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:46.041770 13445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:46.041906 13445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:46.049388 13445 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:46.049496 13445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:46.049666 13445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.71
I20240501 20:10:46.058665 13445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.29.219.71
--webserver_port=0
--tserver_master_addrs=127.29.219.126:34159
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.71
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:46.059965 13445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:46.061676 13445 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:46.073247 13450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:46.073843 13451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:46.074604 13453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:46.074965 13445 server_base.cc:1034] running on GCE node
I20240501 20:10:46.075523 13445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:46.076435 13445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:46.077620 13445 hybrid_clock.cc:648] HybridClock initialized: now 1714594246077571 us; error 36 us; skew 500 ppm
I20240501 20:10:46.077924 13445 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:46.080075 13445 webserver.cc:460] Webserver started at http://127.29.219.71:40565/ using document root <none> and password file <none>
I20240501 20:10:46.080786 13445 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:46.080895 13445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:46.081223 13445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:46.083848 13445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "9053a862fa454257992c47064f1a075d"
format_stamp: "Formatted at 2024-05-01 20:10:46 on dist-test-slave-h2cj"
I20240501 20:10:46.084594 13445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "9053a862fa454257992c47064f1a075d"
format_stamp: "Formatted at 2024-05-01 20:10:46 on dist-test-slave-h2cj"
I20240501 20:10:46.089860 13445 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 20:10:46.093230 13459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:46.094507 13445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240501 20:10:46.094666 13445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "9053a862fa454257992c47064f1a075d"
format_stamp: "Formatted at 2024-05-01 20:10:46 on dist-test-slave-h2cj"
I20240501 20:10:46.094858 13445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:46.121312 13445 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:10:46.122159 13445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:46.122390 13445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:46.123383 13445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:46.124802 13445 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:46.124887 13445 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:46.124967 13445 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:46.125020 13445 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:46.155726 13445 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.71:46811
I20240501 20:10:46.155782 13571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.71:46811 every 8 connection(s)
I20240501 20:10:46.157013 13445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 20:10:46.157543 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 13445
I20240501 20:10:46.157703 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 20:10:46.160935 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.72:0
--local_ip_for_outbound_sockets=127.29.219.72
--webserver_interface=127.29.219.72
--webserver_port=0
--tserver_master_addrs=127.29.219.126:34159
--builtin_ntp_servers=127.29.219.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:10:46.170411 13572 heartbeater.cc:347] Connected to a master server at 127.29.219.126:34159
I20240501 20:10:46.170727 13572 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:46.171528 13572 heartbeater.cc:511] Master 127.29.219.126:34159 requested a full tablet report, sending...
I20240501 20:10:46.172927 12605 ts_manager.cc:194] Registered new tserver with Master: 9053a862fa454257992c47064f1a075d (127.29.219.71:46811)
I20240501 20:10:46.173707 12605 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.71:47821
I20240501 20:10:46.250559 13312 heartbeater.cc:503] Master 127.29.219.126:34159 was elected leader, sending a full tablet report...
W20240501 20:10:46.501574 13575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:46.501912 13575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:10:46.502045 13575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:46.509593 13575 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:10:46.509742 13575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:10:46.509903 13575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.72
I20240501 20:10:46.518929 13575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.29.219.72
--webserver_port=0
--tserver_master_addrs=127.29.219.126:34159
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.72
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:10:46.520249 13575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:10:46.522002 13575 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:10:46.533165 13581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:46.533892 13580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:10:46.534212 13583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:10:46.534443 13575 server_base.cc:1034] running on GCE node
I20240501 20:10:46.535131 13575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:10:46.536027 13575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:10:46.537235 13575 hybrid_clock.cc:648] HybridClock initialized: now 1714594246537178 us; error 50 us; skew 500 ppm
I20240501 20:10:46.537533 13575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:10:46.539727 13575 webserver.cc:460] Webserver started at http://127.29.219.72:37019/ using document root <none> and password file <none>
I20240501 20:10:46.540438 13575 fs_manager.cc:359] Metadata directory not provided
I20240501 20:10:46.540546 13575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:10:46.540861 13575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 20:10:46.543519 13575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "078b7bc667e2484f9f25124462c84da9"
format_stamp: "Formatted at 2024-05-01 20:10:46 on dist-test-slave-h2cj"
I20240501 20:10:46.544322 13575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "078b7bc667e2484f9f25124462c84da9"
format_stamp: "Formatted at 2024-05-01 20:10:46 on dist-test-slave-h2cj"
I20240501 20:10:46.549460 13575 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 20:10:46.552744 13589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:46.553977 13575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:46.554136 13575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "078b7bc667e2484f9f25124462c84da9"
format_stamp: "Formatted at 2024-05-01 20:10:46 on dist-test-slave-h2cj"
I20240501 20:10:46.554329 13575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:10:46.571687 13575 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:10:46.572546 13575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:10:46.572806 13575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:10:46.573756 13575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:10:46.575289 13575 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:10:46.575385 13575 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:46.575510 13575 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:10:46.575587 13575 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:10:46.606153 13575 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.72:33839
I20240501 20:10:46.606201 13701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.72:33839 every 8 connection(s)
I20240501 20:10:46.607437 13575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 20:10:46.615690 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 13575
I20240501 20:10:46.615867 30573 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 20:10:46.619663 13702 heartbeater.cc:347] Connected to a master server at 127.29.219.126:34159
I20240501 20:10:46.619930 13702 heartbeater.cc:464] Registering TS with master...
I20240501 20:10:46.620676 13702 heartbeater.cc:511] Master 127.29.219.126:34159 requested a full tablet report, sending...
I20240501 20:10:46.621920 12604 ts_manager.cc:194] Registered new tserver with Master: 078b7bc667e2484f9f25124462c84da9 (127.29.219.72:33839)
I20240501 20:10:46.622851 12604 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.72:43571
I20240501 20:10:46.632131 30573 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 20:10:46.648957 30573 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 20:10:46.649080 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 13185
I20240501 20:10:46.662137 13307 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:46.703954 13442 heartbeater.cc:503] Master 127.29.219.126:34159 was elected leader, sending a full tablet report...
I20240501 20:10:46.743913 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 13185
I20240501 20:10:46.760869 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 13315
I20240501 20:10:46.769500 13437 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:46.852631 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 13315
I20240501 20:10:46.872651 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 13445
I20240501 20:10:46.880776 13567 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:46.961356 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 13445
I20240501 20:10:46.977720 30573 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu pid 13575
I20240501 20:10:46.986153 13697 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 20:10:47.067265 30573 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu with pid 13575
I20240501 20:10:50.835906 12604 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52990:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:50.881635 12986 tablet_service.cc:1467] Processing CreateTablet for tablet d85b0f46583649e9bee3f433ba6f928e (DEFAULT_TABLE table=rebalance_test_table_0 [id=343ee7439a3c45b4adc030fabb2992a7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:50.881848 12987 tablet_service.cc:1467] Processing CreateTablet for tablet 50561f1e0c5b47a78abb1fba84e725fa (DEFAULT_TABLE table=rebalance_test_table_0 [id=343ee7439a3c45b4adc030fabb2992a7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:50.883371 12986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d85b0f46583649e9bee3f433ba6f928e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:50.884119 12987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50561f1e0c5b47a78abb1fba84e725fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:50.887615 13117 tablet_service.cc:1467] Processing CreateTablet for tablet 50561f1e0c5b47a78abb1fba84e725fa (DEFAULT_TABLE table=rebalance_test_table_0 [id=343ee7439a3c45b4adc030fabb2992a7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:50.889055 13116 tablet_service.cc:1467] Processing CreateTablet for tablet 5f728b714ca142648ba5aa3459f7ae80 (DEFAULT_TABLE table=rebalance_test_table_0 [id=343ee7439a3c45b4adc030fabb2992a7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:50.889494 13117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50561f1e0c5b47a78abb1fba84e725fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:50.890074 13116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f728b714ca142648ba5aa3459f7ae80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:50.892567 12724 tablet_service.cc:1467] Processing CreateTablet for tablet 5f728b714ca142648ba5aa3459f7ae80 (DEFAULT_TABLE table=rebalance_test_table_0 [id=343ee7439a3c45b4adc030fabb2992a7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:50.895120 12724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f728b714ca142648ba5aa3459f7ae80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:50.894794 12857 tablet_service.cc:1467] Processing CreateTablet for tablet d85b0f46583649e9bee3f433ba6f928e (DEFAULT_TABLE table=rebalance_test_table_0 [id=343ee7439a3c45b4adc030fabb2992a7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:50.896145 12857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d85b0f46583649e9bee3f433ba6f928e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:50.897082 12725 tablet_service.cc:1467] Processing CreateTablet for tablet d85b0f46583649e9bee3f433ba6f928e (DEFAULT_TABLE table=rebalance_test_table_0 [id=343ee7439a3c45b4adc030fabb2992a7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:50.897208 12856 tablet_service.cc:1467] Processing CreateTablet for tablet 5f728b714ca142648ba5aa3459f7ae80 (DEFAULT_TABLE table=rebalance_test_table_0 [id=343ee7439a3c45b4adc030fabb2992a7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:50.897645 12725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d85b0f46583649e9bee3f433ba6f928e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:50.897686 12856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f728b714ca142648ba5aa3459f7ae80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:50.900035 12726 tablet_service.cc:1467] Processing CreateTablet for tablet 50561f1e0c5b47a78abb1fba84e725fa (DEFAULT_TABLE table=rebalance_test_table_0 [id=343ee7439a3c45b4adc030fabb2992a7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:50.900508 12726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50561f1e0c5b47a78abb1fba84e725fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:50.927851 13733 tablet_bootstrap.cc:492] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c: Bootstrap starting.
I20240501 20:10:50.933964 13733 tablet_bootstrap.cc:654] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:50.936311 13731 tablet_bootstrap.cc:492] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362: Bootstrap starting.
I20240501 20:10:50.936307 13733 log.cc:824] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c: Log is configured to *not* fsync() on all Append() calls
I20240501 20:10:50.938542 13735 tablet_bootstrap.cc:492] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62: Bootstrap starting.
I20240501 20:10:50.941231 13731 tablet_bootstrap.cc:654] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:50.944489 13732 tablet_bootstrap.cc:492] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231: Bootstrap starting.
I20240501 20:10:50.945035 13735 tablet_bootstrap.cc:654] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:50.945308 13731 log.cc:824] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362: Log is configured to *not* fsync() on all Append() calls
I20240501 20:10:50.947263 13735 log.cc:824] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62: Log is configured to *not* fsync() on all Append() calls
I20240501 20:10:50.948376 13732 tablet_bootstrap.cc:654] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:50.948773 13731 tablet_bootstrap.cc:492] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362: No bootstrap required, opened a new log
I20240501 20:10:50.948980 13733 tablet_bootstrap.cc:492] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c: No bootstrap required, opened a new log
I20240501 20:10:50.949177 13731 ts_tablet_manager.cc:1367] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20240501 20:10:50.949268 13733 ts_tablet_manager.cc:1367] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.010s
I20240501 20:10:50.949575 13732 log.cc:824] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231: Log is configured to *not* fsync() on all Append() calls
I20240501 20:10:50.959470 13732 tablet_bootstrap.cc:492] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231: No bootstrap required, opened a new log
I20240501 20:10:50.959885 13732 ts_tablet_manager.cc:1367] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20240501 20:10:50.960148 13733 raft_consensus.cc:348] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:50.960721 13733 raft_consensus.cc:374] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:50.960861 13733 raft_consensus.cc:729] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Initialized, Role: FOLLOWER
I20240501 20:10:50.961735 13733 consensus_queue.cc:260] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:50.964280 13733 ts_tablet_manager.cc:1398] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20240501 20:10:50.964056 13731 raft_consensus.cc:348] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:50.964530 13731 raft_consensus.cc:374] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:50.964658 13731 raft_consensus.cc:729] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Initialized, Role: FOLLOWER
I20240501 20:10:50.964730 13733 tablet_bootstrap.cc:492] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c: Bootstrap starting.
I20240501 20:10:50.965340 13731 consensus_queue.cc:260] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:50.966444 13733 tablet_bootstrap.cc:654] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:50.966574 13735 tablet_bootstrap.cc:492] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62: No bootstrap required, opened a new log
I20240501 20:10:50.966920 13735 ts_tablet_manager.cc:1367] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.012s
I20240501 20:10:50.973280 13732 raft_consensus.cc:348] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:50.973959 13732 raft_consensus.cc:374] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:50.974121 13732 raft_consensus.cc:729] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Initialized, Role: FOLLOWER
I20240501 20:10:50.977615 13732 consensus_queue.cc:260] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:50.978439 13731 ts_tablet_manager.cc:1398] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362: Time spent starting tablet: real 0.029s user 0.016s sys 0.010s
I20240501 20:10:50.978964 13731 tablet_bootstrap.cc:492] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362: Bootstrap starting.
I20240501 20:10:50.980268 13733 tablet_bootstrap.cc:492] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c: No bootstrap required, opened a new log
I20240501 20:10:50.980716 13731 tablet_bootstrap.cc:654] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:50.981477 13733 ts_tablet_manager.cc:1367] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.000s
I20240501 20:10:50.982900 13733 raft_consensus.cc:348] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:50.983175 13733 raft_consensus.cc:374] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:50.983480 13733 raft_consensus.cc:729] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Initialized, Role: FOLLOWER
I20240501 20:10:50.983762 13733 consensus_queue.cc:260] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:50.984423 13733 ts_tablet_manager.cc:1398] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:50.984889 13732 ts_tablet_manager.cc:1398] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231: Time spent starting tablet: real 0.025s user 0.013s sys 0.005s
I20240501 20:10:50.985435 13732 tablet_bootstrap.cc:492] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231: Bootstrap starting.
I20240501 20:10:50.987250 13732 tablet_bootstrap.cc:654] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:50.997346 13732 tablet_bootstrap.cc:492] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231: No bootstrap required, opened a new log
I20240501 20:10:50.997673 13732 ts_tablet_manager.cc:1367] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20240501 20:10:50.998109 13735 raft_consensus.cc:348] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:50.998921 13732 raft_consensus.cc:348] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:50.999161 13732 raft_consensus.cc:374] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:50.999315 13732 raft_consensus.cc:729] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Initialized, Role: FOLLOWER
I20240501 20:10:50.999339 13735 raft_consensus.cc:374] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:50.999608 13735 raft_consensus.cc:729] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Initialized, Role: FOLLOWER
I20240501 20:10:50.999641 13732 consensus_queue.cc:260] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:51.000465 13732 ts_tablet_manager.cc:1398] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:51.001549 13735 consensus_queue.cc:260] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:51.003639 13731 tablet_bootstrap.cc:492] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362: No bootstrap required, opened a new log
I20240501 20:10:51.003804 13731 ts_tablet_manager.cc:1367] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362: Time spent bootstrapping tablet: real 0.025s user 0.017s sys 0.005s
I20240501 20:10:51.004246 13735 ts_tablet_manager.cc:1398] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62: Time spent starting tablet: real 0.037s user 0.012s sys 0.003s
I20240501 20:10:51.004778 13731 raft_consensus.cc:348] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:51.004979 13735 tablet_bootstrap.cc:492] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62: Bootstrap starting.
I20240501 20:10:51.005069 13731 raft_consensus.cc:374] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:51.005201 13731 raft_consensus.cc:729] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Initialized, Role: FOLLOWER
I20240501 20:10:51.005486 13731 consensus_queue.cc:260] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:51.006511 13731 ts_tablet_manager.cc:1398] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 20:10:51.007545 13735 tablet_bootstrap.cc:654] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:51.009739 13735 tablet_bootstrap.cc:492] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62: No bootstrap required, opened a new log
I20240501 20:10:51.009886 13735 ts_tablet_manager.cc:1367] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 20:10:51.010725 13735 raft_consensus.cc:348] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:51.010886 13735 raft_consensus.cc:374] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:51.010972 13735 raft_consensus.cc:729] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Initialized, Role: FOLLOWER
I20240501 20:10:51.011174 13735 consensus_queue.cc:260] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:51.011565 13735 ts_tablet_manager.cc:1398] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:51.011794 13735 tablet_bootstrap.cc:492] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62: Bootstrap starting.
I20240501 20:10:51.012904 13735 tablet_bootstrap.cc:654] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:51.014791 13735 tablet_bootstrap.cc:492] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62: No bootstrap required, opened a new log
I20240501 20:10:51.014945 13735 ts_tablet_manager.cc:1367] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240501 20:10:51.015563 13735 raft_consensus.cc:348] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:51.015802 13735 raft_consensus.cc:374] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:51.015874 13735 raft_consensus.cc:729] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Initialized, Role: FOLLOWER
I20240501 20:10:51.016125 13735 consensus_queue.cc:260] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:51.016618 13735 ts_tablet_manager.cc:1398] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240501 20:10:51.023676 13183 tablet.cc:2369] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:10:51.061724 13053 tablet.cc:2369] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 20:10:51.102789 12923 tablet.cc:2369] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:10:51.130131 13745 raft_consensus.cc:482] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:51.130407 13745 raft_consensus.cc:504] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:51.132198 13745 leader_election.cc:288] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:10:51.139242 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f728b714ca142648ba5aa3459f7ae80" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231" is_pre_election: true
I20240501 20:10:51.139582 12877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f728b714ca142648ba5aa3459f7ae80" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c" is_pre_election: true
I20240501 20:10:51.139750 13137 raft_consensus.cc:2454] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 0.
I20240501 20:10:51.140100 12877 raft_consensus.cc:2454] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 0.
I20240501 20:10:51.140431 12681 leader_election.cc:302] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:10:51.140869 13745 raft_consensus.cc:2781] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:51.141041 13745 raft_consensus.cc:482] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:51.141158 13745 raft_consensus.cc:3037] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:51.143304 13745 raft_consensus.cc:504] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:51.143867 13745 leader_election.cc:288] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:10:51.144299 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f728b714ca142648ba5aa3459f7ae80" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:10:51.144371 12877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f728b714ca142648ba5aa3459f7ae80" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:10:51.144474 13137 raft_consensus.cc:3037] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:51.144589 12877 raft_consensus.cc:3037] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:51.146795 12877 raft_consensus.cc:2454] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 1.
I20240501 20:10:51.146812 13137 raft_consensus.cc:2454] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 1.
I20240501 20:10:51.147178 12681 leader_election.cc:302] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters:
I20240501 20:10:51.147461 13745 raft_consensus.cc:2781] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:51.147858 13745 raft_consensus.cc:686] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:10:51.148367 13745 consensus_queue.cc:237] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
W20240501 20:10:51.149791 12792 tablet.cc:2369] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 20:10:51.153182 12605 catalog_manager.cc:5484] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:51.156406 13745 raft_consensus.cc:482] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:51.156605 13745 raft_consensus.cc:504] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:51.157217 13745 leader_election.cc:288] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:10:51.158026 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50561f1e0c5b47a78abb1fba84e725fa" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231" is_pre_election: true
I20240501 20:10:51.158278 13137 raft_consensus.cc:2454] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 0.
I20240501 20:10:51.158684 12681 leader_election.cc:302] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:10:51.159001 13745 raft_consensus.cc:2781] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:51.159094 13745 raft_consensus.cc:482] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:51.159202 13745 raft_consensus.cc:3037] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:51.161154 13745 raft_consensus.cc:504] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:51.161793 13745 leader_election.cc:288] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:10:51.162264 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50561f1e0c5b47a78abb1fba84e725fa" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:10:51.162490 13137 raft_consensus.cc:3037] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:51.163367 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50561f1e0c5b47a78abb1fba84e725fa" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:10:51.163389 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50561f1e0c5b47a78abb1fba84e725fa" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362" is_pre_election: true
I20240501 20:10:51.163990 13007 raft_consensus.cc:2454] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 0.
I20240501 20:10:51.164557 13137 raft_consensus.cc:2454] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 1.
I20240501 20:10:51.164866 12681 leader_election.cc:302] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, bb1be87a7aee48a4b405b9d209c61231; no voters: b4720929b44c4560bfdf08107e137362
I20240501 20:10:51.165163 13745 raft_consensus.cc:2781] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:51.165323 13745 raft_consensus.cc:686] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:10:51.165614 13745 consensus_queue.cc:237] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:51.168006 12605 catalog_manager.cc:5484] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:51.174165 13739 raft_consensus.cc:482] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:51.174345 13739 raft_consensus.cc:504] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:51.175936 13739 leader_election.cc:288] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:10:51.182809 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85b0f46583649e9bee3f433ba6f928e" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362" is_pre_election: true
I20240501 20:10:51.183030 13007 raft_consensus.cc:2454] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 0.
I20240501 20:10:51.182920 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85b0f46583649e9bee3f433ba6f928e" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9d799e45f8407590cb3096f5625c62" is_pre_election: true
I20240501 20:10:51.183358 12746 raft_consensus.cc:2454] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 0.
I20240501 20:10:51.183495 12811 leader_election.cc:302] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:10:51.183895 13739 raft_consensus.cc:2781] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:51.184044 13739 raft_consensus.cc:482] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:51.184159 13739 raft_consensus.cc:3037] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:51.186139 13739 raft_consensus.cc:504] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:51.186779 13739 leader_election.cc:288] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:10:51.187242 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85b0f46583649e9bee3f433ba6f928e" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:10:51.187306 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85b0f46583649e9bee3f433ba6f928e" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:10:51.187433 12746 raft_consensus.cc:3037] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:51.187510 13007 raft_consensus.cc:3037] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:51.189535 12746 raft_consensus.cc:2454] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 1.
I20240501 20:10:51.189543 13007 raft_consensus.cc:2454] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 1.
I20240501 20:10:51.189967 12811 leader_election.cc:302] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:10:51.190271 13739 raft_consensus.cc:2781] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:51.190709 13739 raft_consensus.cc:686] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:10:51.191251 13739 consensus_queue.cc:237] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:51.196048 12605 catalog_manager.cc:5484] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 1 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } }
W20240501 20:10:51.544016 13757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:51.544324 13757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 20:10:51.550956 13745 consensus_queue.cc:1035] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:10:51.551805 13757 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:10:51.558422 13007 raft_consensus.cc:3037] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:51.562496 13745 consensus_queue.cc:1035] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:51.571887 13745 consensus_queue.cc:1035] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:51.579432 13745 consensus_queue.cc:1035] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
Using table 'rebalance_test_table_0'
I20240501 20:10:51.701262 13007 raft_consensus.cc:1261] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Refusing update from remote peer 77bf705599644326b6e71b5caf32fc7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:51.702148 13739 consensus_queue.cc:1035] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:51.702880 12746 raft_consensus.cc:1261] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Refusing update from remote peer 77bf705599644326b6e71b5caf32fc7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 20:10:51.705425 13739 consensus_queue.cc:1035] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:51.800433 13794 mvcc.cc:205] Tried to move back new op lower bound from 7022978054957096960 to 7022978052883263488. Current Snapshot: MvccSnapshot[applied={T|T < 7022978054957096960}]
INSERT report
rows total: 2000
time total: 252.864 ms
time per row: 0.126432 ms
I20240501 20:10:51.962800 12603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52990:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:51.971535 12986 tablet_service.cc:1467] Processing CreateTablet for tablet 7c3933feed404a4baee16428d2aafacf (DEFAULT_TABLE table=rebalance_test_table_1 [id=05f8b6130a4a454d90f5d713be5b7c0c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:51.971784 12857 tablet_service.cc:1467] Processing CreateTablet for tablet 7c3933feed404a4baee16428d2aafacf (DEFAULT_TABLE table=rebalance_test_table_1 [id=05f8b6130a4a454d90f5d713be5b7c0c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:51.972038 12986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c3933feed404a4baee16428d2aafacf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:51.972227 12857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c3933feed404a4baee16428d2aafacf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:51.972505 12856 tablet_service.cc:1467] Processing CreateTablet for tablet a8b101e9e38b4048bc86d76f97201534 (DEFAULT_TABLE table=rebalance_test_table_1 [id=05f8b6130a4a454d90f5d713be5b7c0c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:51.972853 12856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8b101e9e38b4048bc86d76f97201534. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:51.973501 12726 tablet_service.cc:1467] Processing CreateTablet for tablet 4ddb529dd5534f8bb66562e8467c1f12 (DEFAULT_TABLE table=rebalance_test_table_1 [id=05f8b6130a4a454d90f5d713be5b7c0c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:51.973924 12726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ddb529dd5534f8bb66562e8467c1f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:51.975376 12855 tablet_service.cc:1467] Processing CreateTablet for tablet 4ddb529dd5534f8bb66562e8467c1f12 (DEFAULT_TABLE table=rebalance_test_table_1 [id=05f8b6130a4a454d90f5d713be5b7c0c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:51.975786 12855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ddb529dd5534f8bb66562e8467c1f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:51.978780 13116 tablet_service.cc:1467] Processing CreateTablet for tablet 7c3933feed404a4baee16428d2aafacf (DEFAULT_TABLE table=rebalance_test_table_1 [id=05f8b6130a4a454d90f5d713be5b7c0c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:51.979207 13116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c3933feed404a4baee16428d2aafacf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:51.980517 13117 tablet_service.cc:1467] Processing CreateTablet for tablet a8b101e9e38b4048bc86d76f97201534 (DEFAULT_TABLE table=rebalance_test_table_1 [id=05f8b6130a4a454d90f5d713be5b7c0c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:51.980916 13117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8b101e9e38b4048bc86d76f97201534. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:51.983708 12987 tablet_service.cc:1467] Processing CreateTablet for tablet a8b101e9e38b4048bc86d76f97201534 (DEFAULT_TABLE table=rebalance_test_table_1 [id=05f8b6130a4a454d90f5d713be5b7c0c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:51.984275 12987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8b101e9e38b4048bc86d76f97201534. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:51.985116 13825 tablet_bootstrap.cc:492] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62: Bootstrap starting.
I20240501 20:10:51.983759 12985 tablet_service.cc:1467] Processing CreateTablet for tablet 4ddb529dd5534f8bb66562e8467c1f12 (DEFAULT_TABLE table=rebalance_test_table_1 [id=05f8b6130a4a454d90f5d713be5b7c0c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:51.987090 12985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ddb529dd5534f8bb66562e8467c1f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:51.988008 13825 tablet_bootstrap.cc:654] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:51.991242 13826 tablet_bootstrap.cc:492] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231: Bootstrap starting.
I20240501 20:10:51.994769 13826 tablet_bootstrap.cc:654] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:51.995929 13825 tablet_bootstrap.cc:492] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62: No bootstrap required, opened a new log
I20240501 20:10:51.996237 13825 ts_tablet_manager.cc:1367] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.001s
I20240501 20:10:51.998570 13823 tablet_bootstrap.cc:492] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362: Bootstrap starting.
I20240501 20:10:51.997362 13825 raft_consensus.cc:348] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:51.999280 13825 raft_consensus.cc:374] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:51.999480 13825 raft_consensus.cc:729] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Initialized, Role: FOLLOWER
I20240501 20:10:51.999982 13825 consensus_queue.cc:260] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:52.001142 13825 ts_tablet_manager.cc:1398] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 20:10:52.001220 13823 tablet_bootstrap.cc:654] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:52.003262 13824 tablet_bootstrap.cc:492] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c: Bootstrap starting.
I20240501 20:10:52.004211 13826 tablet_bootstrap.cc:492] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231: No bootstrap required, opened a new log
I20240501 20:10:52.004369 13826 ts_tablet_manager.cc:1367] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20240501 20:10:52.005107 13824 tablet_bootstrap.cc:654] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:52.005287 13826 raft_consensus.cc:348] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:52.005484 13826 raft_consensus.cc:374] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:52.005570 13826 raft_consensus.cc:729] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Initialized, Role: FOLLOWER
I20240501 20:10:52.005793 13826 consensus_queue.cc:260] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:52.006385 13826 ts_tablet_manager.cc:1398] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:10:52.006690 13826 tablet_bootstrap.cc:492] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231: Bootstrap starting.
I20240501 20:10:52.008514 13826 tablet_bootstrap.cc:654] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:52.011715 13824 tablet_bootstrap.cc:492] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c: No bootstrap required, opened a new log
I20240501 20:10:52.011994 13824 ts_tablet_manager.cc:1367] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20240501 20:10:52.013065 13824 raft_consensus.cc:348] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:52.013373 13824 raft_consensus.cc:374] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:52.013486 13824 raft_consensus.cc:729] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Initialized, Role: FOLLOWER
I20240501 20:10:52.013670 13826 tablet_bootstrap.cc:492] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231: No bootstrap required, opened a new log
I20240501 20:10:52.013832 13826 ts_tablet_manager.cc:1367] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20240501 20:10:52.013731 13824 consensus_queue.cc:260] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:52.013891 13823 tablet_bootstrap.cc:492] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362: No bootstrap required, opened a new log
I20240501 20:10:52.014077 13823 ts_tablet_manager.cc:1367] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.004s
I20240501 20:10:52.014320 13824 ts_tablet_manager.cc:1398] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:10:52.014716 13824 tablet_bootstrap.cc:492] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c: Bootstrap starting.
I20240501 20:10:52.014644 13826 raft_consensus.cc:348] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:52.014894 13826 raft_consensus.cc:374] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:52.014981 13826 raft_consensus.cc:729] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Initialized, Role: FOLLOWER
I20240501 20:10:52.015004 13823 raft_consensus.cc:348] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:52.015197 13823 raft_consensus.cc:374] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:52.015295 13823 raft_consensus.cc:729] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Initialized, Role: FOLLOWER
I20240501 20:10:52.015242 13826 consensus_queue.cc:260] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:52.015776 13826 ts_tablet_manager.cc:1398] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:10:52.016086 13823 consensus_queue.cc:260] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:52.016669 13823 ts_tablet_manager.cc:1398] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:52.017166 13823 tablet_bootstrap.cc:492] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362: Bootstrap starting.
I20240501 20:10:52.018867 13824 tablet_bootstrap.cc:654] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:52.019910 13823 tablet_bootstrap.cc:654] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:52.023921 13824 tablet_bootstrap.cc:492] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c: No bootstrap required, opened a new log
I20240501 20:10:52.023953 13823 tablet_bootstrap.cc:492] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362: No bootstrap required, opened a new log
I20240501 20:10:52.024113 13823 ts_tablet_manager.cc:1367] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 20:10:52.024121 13824 ts_tablet_manager.cc:1367] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20240501 20:10:52.024904 13823 raft_consensus.cc:348] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:52.024904 13824 raft_consensus.cc:348] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:52.025144 13823 raft_consensus.cc:374] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:52.025144 13824 raft_consensus.cc:374] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:52.025230 13823 raft_consensus.cc:729] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Initialized, Role: FOLLOWER
I20240501 20:10:52.025230 13824 raft_consensus.cc:729] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Initialized, Role: FOLLOWER
I20240501 20:10:52.025485 13824 consensus_queue.cc:260] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:52.025485 13823 consensus_queue.cc:260] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:52.026018 13823 ts_tablet_manager.cc:1398] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:52.026556 13824 ts_tablet_manager.cc:1398] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 20:10:52.026882 13824 tablet_bootstrap.cc:492] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c: Bootstrap starting.
I20240501 20:10:52.027171 13823 tablet_bootstrap.cc:492] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362: Bootstrap starting.
I20240501 20:10:52.028573 13824 tablet_bootstrap.cc:654] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:52.028829 13823 tablet_bootstrap.cc:654] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:52.031728 13824 tablet_bootstrap.cc:492] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c: No bootstrap required, opened a new log
I20240501 20:10:52.031880 13824 ts_tablet_manager.cc:1367] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 20:10:52.032122 13823 tablet_bootstrap.cc:492] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362: No bootstrap required, opened a new log
I20240501 20:10:52.032260 13823 ts_tablet_manager.cc:1367] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:10:52.032717 13824 raft_consensus.cc:348] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:52.032919 13824 raft_consensus.cc:374] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:52.033032 13824 raft_consensus.cc:729] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Initialized, Role: FOLLOWER
I20240501 20:10:52.032984 13823 raft_consensus.cc:348] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:52.033181 13823 raft_consensus.cc:374] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:52.033274 13823 raft_consensus.cc:729] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Initialized, Role: FOLLOWER
I20240501 20:10:52.033294 13824 consensus_queue.cc:260] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:52.033490 13823 consensus_queue.cc:260] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:52.033844 13824 ts_tablet_manager.cc:1398] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:52.034018 13823 ts_tablet_manager.cc:1398] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:52.097216 13765 raft_consensus.cc:482] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:52.097432 13765 raft_consensus.cc:504] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:52.099074 13765 leader_election.cc:288] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:10:52.105597 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c3933feed404a4baee16428d2aafacf" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c" is_pre_election: true
I20240501 20:10:52.105863 12876 raft_consensus.cc:2454] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 0.
I20240501 20:10:52.106315 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c3933feed404a4baee16428d2aafacf" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231" is_pre_election: true
I20240501 20:10:52.106361 12942 leader_election.cc:302] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:10:52.106505 13136 raft_consensus.cc:2454] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 0.
I20240501 20:10:52.106798 13765 raft_consensus.cc:2781] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:52.106928 13765 raft_consensus.cc:482] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:52.107033 13765 raft_consensus.cc:3037] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:52.109277 13765 raft_consensus.cc:504] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:52.109838 13765 leader_election.cc:288] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:10:52.110337 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c3933feed404a4baee16428d2aafacf" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:10:52.110352 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c3933feed404a4baee16428d2aafacf" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:10:52.110538 13136 raft_consensus.cc:3037] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:52.110576 12876 raft_consensus.cc:3037] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:52.112453 13136 raft_consensus.cc:2454] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 1.
I20240501 20:10:52.112709 12876 raft_consensus.cc:2454] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 1.
I20240501 20:10:52.112869 12942 leader_election.cc:302] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:10:52.113245 13765 raft_consensus.cc:2781] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:52.113795 13765 raft_consensus.cc:686] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 1 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:10:52.114280 13765 consensus_queue.cc:237] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:52.118101 12600 catalog_manager.cc:5484] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 1 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:52.123328 13813 raft_consensus.cc:482] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:52.123498 13813 raft_consensus.cc:504] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:52.124403 13813 leader_election.cc:288] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:10:52.124943 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b101e9e38b4048bc86d76f97201534" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362" is_pre_election: true
I20240501 20:10:52.125288 13006 raft_consensus.cc:2454] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 0.
I20240501 20:10:52.125764 12811 leader_election.cc:302] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:10:52.126046 13813 raft_consensus.cc:2781] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:52.126163 13813 raft_consensus.cc:482] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:52.126259 13813 raft_consensus.cc:3037] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:52.128314 13813 raft_consensus.cc:504] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:52.129268 13813 leader_election.cc:288] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:10:52.129421 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b101e9e38b4048bc86d76f97201534" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:10:52.129654 13006 raft_consensus.cc:3037] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:52.131664 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b101e9e38b4048bc86d76f97201534" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231" is_pre_election: true
I20240501 20:10:52.131742 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b101e9e38b4048bc86d76f97201534" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:10:52.131934 13006 raft_consensus.cc:2454] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 1.
I20240501 20:10:52.131955 13136 raft_consensus.cc:2454] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 0.
I20240501 20:10:52.132329 12811 leader_election.cc:302] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:10:52.132670 13813 raft_consensus.cc:2781] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:52.132815 13813 raft_consensus.cc:686] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:10:52.133095 13813 consensus_queue.cc:237] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:52.135598 12600 catalog_manager.cc:5484] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 1 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:52.163865 13813 raft_consensus.cc:482] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:52.164113 13813 raft_consensus.cc:504] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:52.164736 13813 leader_election.cc:288] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:10:52.165367 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ddb529dd5534f8bb66562e8467c1f12" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9d799e45f8407590cb3096f5625c62" is_pre_election: true
I20240501 20:10:52.165367 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ddb529dd5534f8bb66562e8467c1f12" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362" is_pre_election: true
I20240501 20:10:52.165613 12746 raft_consensus.cc:2454] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 0.
I20240501 20:10:52.165614 13006 raft_consensus.cc:2454] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 0.
I20240501 20:10:52.166062 12812 leader_election.cc:302] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters:
I20240501 20:10:52.166334 13813 raft_consensus.cc:2781] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:52.166415 13813 raft_consensus.cc:482] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:52.166517 13813 raft_consensus.cc:3037] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:52.168417 13813 raft_consensus.cc:504] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:52.168938 13813 leader_election.cc:288] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:10:52.169385 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ddb529dd5534f8bb66562e8467c1f12" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:10:52.169438 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ddb529dd5534f8bb66562e8467c1f12" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:10:52.169554 12746 raft_consensus.cc:3037] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:52.169636 13006 raft_consensus.cc:3037] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:52.171489 12746 raft_consensus.cc:2454] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 1.
I20240501 20:10:52.171581 13006 raft_consensus.cc:2454] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 1.
I20240501 20:10:52.171907 12812 leader_election.cc:302] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters:
I20240501 20:10:52.172209 13813 raft_consensus.cc:2781] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:52.172356 13813 raft_consensus.cc:686] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:10:52.172623 13813 consensus_queue.cc:237] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:52.175417 12600 catalog_manager.cc:5484] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 1 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } }
W20240501 20:10:52.528849 13844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:52.529232 13844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:52.537166 13844 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 20:10:52.598160 13765 consensus_queue.cc:1035] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:52.614094 13839 consensus_queue.cc:1035] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:52.617810 13137 raft_consensus.cc:3037] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:52.621961 13813 consensus_queue.cc:1035] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:52.636278 13813 consensus_queue.cc:1035] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:52.655989 13801 consensus_queue.cc:1035] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:52.684895 13813 consensus_queue.cc:1035] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:10:52.760680 12681 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.66:32867, user_credentials={real_user=slave}} blocked reactor thread for 55279us
INSERT report
rows total: 2000
time total: 249.464 ms
time per row: 0.124732 ms
I20240501 20:10:52.955317 12602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52990:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:52.964426 13116 tablet_service.cc:1467] Processing CreateTablet for tablet 248dfbfcb6214244a410083e8de22743 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c7bc2b355e54a6f89825bf892ed7e1a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:52.964682 12856 tablet_service.cc:1467] Processing CreateTablet for tablet 248dfbfcb6214244a410083e8de22743 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c7bc2b355e54a6f89825bf892ed7e1a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:52.964946 13116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 248dfbfcb6214244a410083e8de22743. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:52.964876 12726 tablet_service.cc:1467] Processing CreateTablet for tablet 248dfbfcb6214244a410083e8de22743 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c7bc2b355e54a6f89825bf892ed7e1a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:52.965057 12856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 248dfbfcb6214244a410083e8de22743. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:52.965356 12726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 248dfbfcb6214244a410083e8de22743. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:52.965590 12987 tablet_service.cc:1467] Processing CreateTablet for tablet 436330ea0516443799906257122b38af (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c7bc2b355e54a6f89825bf892ed7e1a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:52.966078 12987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436330ea0516443799906257122b38af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:52.967475 12725 tablet_service.cc:1467] Processing CreateTablet for tablet 436330ea0516443799906257122b38af (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c7bc2b355e54a6f89825bf892ed7e1a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:52.967842 12725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436330ea0516443799906257122b38af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:52.968915 13117 tablet_service.cc:1467] Processing CreateTablet for tablet 436330ea0516443799906257122b38af (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c7bc2b355e54a6f89825bf892ed7e1a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:52.969295 13117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436330ea0516443799906257122b38af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:52.972558 13115 tablet_service.cc:1467] Processing CreateTablet for tablet e00b9b3645a04a4686e5759bd8201eca (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c7bc2b355e54a6f89825bf892ed7e1a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:52.973109 13115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e00b9b3645a04a4686e5759bd8201eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:52.979723 12724 tablet_service.cc:1467] Processing CreateTablet for tablet e00b9b3645a04a4686e5759bd8201eca (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c7bc2b355e54a6f89825bf892ed7e1a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:52.980320 12724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e00b9b3645a04a4686e5759bd8201eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:52.983927 12855 tablet_service.cc:1467] Processing CreateTablet for tablet e00b9b3645a04a4686e5759bd8201eca (DEFAULT_TABLE table=rebalance_test_table_2 [id=6c7bc2b355e54a6f89825bf892ed7e1a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:52.984416 12855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e00b9b3645a04a4686e5759bd8201eca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:52.984481 13911 tablet_bootstrap.cc:492] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c: Bootstrap starting.
I20240501 20:10:52.987123 13909 tablet_bootstrap.cc:492] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231: Bootstrap starting.
I20240501 20:10:52.989013 13909 tablet_bootstrap.cc:654] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:52.990446 13911 tablet_bootstrap.cc:654] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:52.992463 13909 tablet_bootstrap.cc:492] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231: No bootstrap required, opened a new log
I20240501 20:10:52.992755 13909 ts_tablet_manager.cc:1367] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 20:10:52.995016 13909 raft_consensus.cc:348] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:52.995352 13911 tablet_bootstrap.cc:492] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c: No bootstrap required, opened a new log
I20240501 20:10:52.995388 13909 raft_consensus.cc:374] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:52.995525 13909 raft_consensus.cc:729] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Initialized, Role: FOLLOWER
I20240501 20:10:52.995541 13911 ts_tablet_manager.cc:1367] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.006s
I20240501 20:10:52.995846 13909 consensus_queue.cc:260] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:52.996315 13911 raft_consensus.cc:348] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:52.996555 13911 raft_consensus.cc:374] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:52.996655 13909 ts_tablet_manager.cc:1398] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:10:52.996685 13911 raft_consensus.cc:729] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Initialized, Role: FOLLOWER
I20240501 20:10:52.996978 13909 tablet_bootstrap.cc:492] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231: Bootstrap starting.
I20240501 20:10:52.998603 13909 tablet_bootstrap.cc:654] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:53.003230 13911 consensus_queue.cc:260] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:53.004016 13911 ts_tablet_manager.cc:1398] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240501 20:10:53.004290 13911 tablet_bootstrap.cc:492] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c: Bootstrap starting.
I20240501 20:10:53.006099 13914 tablet_bootstrap.cc:492] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62: Bootstrap starting.
I20240501 20:10:53.006393 13911 tablet_bootstrap.cc:654] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:53.010811 13914 tablet_bootstrap.cc:654] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:53.011510 13909 tablet_bootstrap.cc:492] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231: No bootstrap required, opened a new log
I20240501 20:10:53.011677 13909 ts_tablet_manager.cc:1367] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20240501 20:10:53.012445 13909 raft_consensus.cc:348] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:53.012753 13909 raft_consensus.cc:374] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:53.012789 13910 tablet_bootstrap.cc:492] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362: Bootstrap starting.
I20240501 20:10:53.012879 13909 raft_consensus.cc:729] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Initialized, Role: FOLLOWER
I20240501 20:10:53.013051 13911 tablet_bootstrap.cc:492] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c: No bootstrap required, opened a new log
I20240501 20:10:53.013206 13911 ts_tablet_manager.cc:1367] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.001s
I20240501 20:10:53.013263 13909 consensus_queue.cc:260] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:53.013552 13914 tablet_bootstrap.cc:492] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62: No bootstrap required, opened a new log
I20240501 20:10:53.013706 13914 ts_tablet_manager.cc:1367] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20240501 20:10:53.013959 13909 ts_tablet_manager.cc:1398] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:10:53.013969 13911 raft_consensus.cc:348] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:53.014182 13911 raft_consensus.cc:374] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:53.014257 13911 raft_consensus.cc:729] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Initialized, Role: FOLLOWER
I20240501 20:10:53.014287 13909 tablet_bootstrap.cc:492] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231: Bootstrap starting.
I20240501 20:10:53.014497 13911 consensus_queue.cc:260] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:53.015022 13911 ts_tablet_manager.cc:1398] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:53.016346 13910 tablet_bootstrap.cc:654] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:53.016409 13909 tablet_bootstrap.cc:654] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:53.016535 13914 raft_consensus.cc:348] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:53.016796 13914 raft_consensus.cc:374] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:53.016930 13914 raft_consensus.cc:729] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Initialized, Role: FOLLOWER
I20240501 20:10:53.017207 13914 consensus_queue.cc:260] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:53.017892 13914 ts_tablet_manager.cc:1398] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 20:10:53.019461 13914 tablet_bootstrap.cc:492] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62: Bootstrap starting.
I20240501 20:10:53.021157 13914 tablet_bootstrap.cc:654] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:53.023295 13910 tablet_bootstrap.cc:492] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362: No bootstrap required, opened a new log
I20240501 20:10:53.023396 13914 tablet_bootstrap.cc:492] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62: No bootstrap required, opened a new log
I20240501 20:10:53.023448 13910 ts_tablet_manager.cc:1367] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20240501 20:10:53.023550 13914 ts_tablet_manager.cc:1367] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 20:10:53.024086 13914 raft_consensus.cc:348] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:53.024129 13910 raft_consensus.cc:348] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:53.024260 13914 raft_consensus.cc:374] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:53.024279 13910 raft_consensus.cc:374] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:53.024330 13914 raft_consensus.cc:729] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Initialized, Role: FOLLOWER
I20240501 20:10:53.024358 13910 raft_consensus.cc:729] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Initialized, Role: FOLLOWER
I20240501 20:10:53.024617 13910 consensus_queue.cc:260] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:53.025108 13910 ts_tablet_manager.cc:1398] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:53.025338 13914 consensus_queue.cc:260] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:53.025545 13909 tablet_bootstrap.cc:492] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231: No bootstrap required, opened a new log
I20240501 20:10:53.025769 13909 ts_tablet_manager.cc:1367] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20240501 20:10:53.026010 13914 ts_tablet_manager.cc:1398] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:10:53.026371 13914 tablet_bootstrap.cc:492] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62: Bootstrap starting.
I20240501 20:10:53.026477 13909 raft_consensus.cc:348] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:53.026679 13909 raft_consensus.cc:374] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:53.026777 13909 raft_consensus.cc:729] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Initialized, Role: FOLLOWER
I20240501 20:10:53.027047 13909 consensus_queue.cc:260] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:53.027607 13909 ts_tablet_manager.cc:1398] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:10:53.029467 13914 tablet_bootstrap.cc:654] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:53.032652 13914 tablet_bootstrap.cc:492] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62: No bootstrap required, opened a new log
I20240501 20:10:53.032855 13914 ts_tablet_manager.cc:1367] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 20:10:53.033492 13914 raft_consensus.cc:348] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:53.033691 13914 raft_consensus.cc:374] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:53.033808 13914 raft_consensus.cc:729] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Initialized, Role: FOLLOWER
I20240501 20:10:53.034021 13914 consensus_queue.cc:260] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:53.034552 13914 ts_tablet_manager.cc:1398] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:53.175765 13886 raft_consensus.cc:482] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:53.175992 13886 raft_consensus.cc:504] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:53.176823 13886 leader_election.cc:288] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:10:53.178097 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436330ea0516443799906257122b38af" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231" is_pre_election: true
I20240501 20:10:53.178350 13136 raft_consensus.cc:2454] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 0.
I20240501 20:10:53.178881 12942 leader_election.cc:302] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:10:53.179200 13886 raft_consensus.cc:2781] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:53.179311 13886 raft_consensus.cc:482] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:53.179401 13886 raft_consensus.cc:3037] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:53.181478 13886 raft_consensus.cc:504] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:53.182121 13886 leader_election.cc:288] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:10:53.182572 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436330ea0516443799906257122b38af" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:10:53.182811 13136 raft_consensus.cc:3037] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:53.183540 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436330ea0516443799906257122b38af" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9d799e45f8407590cb3096f5625c62" is_pre_election: true
I20240501 20:10:53.183774 12746 raft_consensus.cc:2454] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 0.
I20240501 20:10:53.183718 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436330ea0516443799906257122b38af" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:10:53.183939 12745 raft_consensus.cc:3037] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:53.185359 13136 raft_consensus.cc:2454] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 1.
I20240501 20:10:53.185812 12942 leader_election.cc:302] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:10:53.186097 13886 raft_consensus.cc:2781] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:53.186236 13886 raft_consensus.cc:686] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 1 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:10:53.186501 13886 consensus_queue.cc:237] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:53.186718 12745 raft_consensus.cc:2454] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 1.
I20240501 20:10:53.189222 12602 catalog_manager.cc:5484] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 1 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:53.197067 13895 raft_consensus.cc:482] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:53.197264 13895 raft_consensus.cc:504] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:53.198006 13895 leader_election.cc:288] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:10:53.198567 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "248dfbfcb6214244a410083e8de22743" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231" is_pre_election: true
I20240501 20:10:53.198604 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "248dfbfcb6214244a410083e8de22743" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9d799e45f8407590cb3096f5625c62" is_pre_election: true
I20240501 20:10:53.198828 13136 raft_consensus.cc:2454] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 0.
I20240501 20:10:53.198887 12745 raft_consensus.cc:2454] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 0.
I20240501 20:10:53.199268 12812 leader_election.cc:302] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters:
I20240501 20:10:53.199535 13895 raft_consensus.cc:2781] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:53.199653 13895 raft_consensus.cc:482] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:53.199743 13895 raft_consensus.cc:3037] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:53.201717 13895 raft_consensus.cc:504] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:53.202282 13895 leader_election.cc:288] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:10:53.202751 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "248dfbfcb6214244a410083e8de22743" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:10:53.202955 13136 raft_consensus.cc:3037] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:53.202908 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "248dfbfcb6214244a410083e8de22743" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:10:53.203127 12745 raft_consensus.cc:3037] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:53.204849 13136 raft_consensus.cc:2454] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 1.
I20240501 20:10:53.204946 12745 raft_consensus.cc:2454] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 1.
I20240501 20:10:53.205243 12812 leader_election.cc:302] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:10:53.205559 13895 raft_consensus.cc:2781] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:53.205708 13895 raft_consensus.cc:686] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:10:53.206007 13895 consensus_queue.cc:237] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:53.208688 12602 catalog_manager.cc:5484] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 1 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:53.259537 13745 raft_consensus.cc:482] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:53.259748 13745 raft_consensus.cc:504] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:53.260355 13745 leader_election.cc:288] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:10:53.260982 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00b9b3645a04a4686e5759bd8201eca" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c" is_pre_election: true
I20240501 20:10:53.260978 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00b9b3645a04a4686e5759bd8201eca" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231" is_pre_election: true
I20240501 20:10:53.261212 12876 raft_consensus.cc:2454] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 0.
I20240501 20:10:53.261231 13137 raft_consensus.cc:2454] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 0.
I20240501 20:10:53.261601 12681 leader_election.cc:302] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:10:53.261931 13745 raft_consensus.cc:2781] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:53.262074 13745 raft_consensus.cc:482] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:53.262168 13745 raft_consensus.cc:3037] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:53.264217 13745 raft_consensus.cc:504] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:53.264734 13745 leader_election.cc:288] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:10:53.265161 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00b9b3645a04a4686e5759bd8201eca" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:10:53.265177 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00b9b3645a04a4686e5759bd8201eca" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:10:53.265354 12876 raft_consensus.cc:3037] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:53.265334 13137 raft_consensus.cc:3037] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:53.267333 12876 raft_consensus.cc:2454] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 1.
I20240501 20:10:53.267341 13137 raft_consensus.cc:2454] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 1.
I20240501 20:10:53.267719 12681 leader_election.cc:302] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters:
I20240501 20:10:53.268034 13745 raft_consensus.cc:2781] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:53.268191 13745 raft_consensus.cc:686] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:10:53.268482 13745 consensus_queue.cc:237] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:53.271196 12602 catalog_manager.cc:5484] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
W20240501 20:10:53.620689 13928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:53.621034 13928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:53.628832 13928 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:10:53.650976 13886 consensus_queue.cc:1035] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:53.659137 13886 consensus_queue.cc:1035] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240501 20:10:53.739161 13745 consensus_queue.cc:1035] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:53.754529 13965 consensus_queue.cc:1035] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:53.787643 13895 consensus_queue.cc:1035] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:10:53.827786 12942 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.68:37951, user_credentials={real_user=slave}} blocked reactor thread for 71218.5us
I20240501 20:10:53.833271 13975 consensus_queue.cc:1035] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 280.304 ms
time per row: 0.140152 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:10:55.012931 12603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52990:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:55.022336 12987 tablet_service.cc:1467] Processing CreateTablet for tablet 47b8c200bb05478db26c0d10b25a514c (DEFAULT_TABLE table=rebalance_test_table_3 [id=0f0d1e3fbb7e4ab888b8cadb04bf3b61]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:55.022892 12987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47b8c200bb05478db26c0d10b25a514c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:55.022920 12855 tablet_service.cc:1467] Processing CreateTablet for tablet 47b8c200bb05478db26c0d10b25a514c (DEFAULT_TABLE table=rebalance_test_table_3 [id=0f0d1e3fbb7e4ab888b8cadb04bf3b61]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:55.023109 12726 tablet_service.cc:1467] Processing CreateTablet for tablet 47b8c200bb05478db26c0d10b25a514c (DEFAULT_TABLE table=rebalance_test_table_3 [id=0f0d1e3fbb7e4ab888b8cadb04bf3b61]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:55.023252 12855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47b8c200bb05478db26c0d10b25a514c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:55.023507 12726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47b8c200bb05478db26c0d10b25a514c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:55.027643 12724 tablet_service.cc:1467] Processing CreateTablet for tablet 70cd215b4f9445499e14c9fd18f7f7fe (DEFAULT_TABLE table=rebalance_test_table_3 [id=0f0d1e3fbb7e4ab888b8cadb04bf3b61]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:55.027652 12985 tablet_service.cc:1467] Processing CreateTablet for tablet 70cd215b4f9445499e14c9fd18f7f7fe (DEFAULT_TABLE table=rebalance_test_table_3 [id=0f0d1e3fbb7e4ab888b8cadb04bf3b61]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:55.028039 12724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70cd215b4f9445499e14c9fd18f7f7fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:55.028172 12985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70cd215b4f9445499e14c9fd18f7f7fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:55.037827 14001 tablet_bootstrap.cc:492] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362: Bootstrap starting.
I20240501 20:10:55.040206 14001 tablet_bootstrap.cc:654] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:55.041163 13115 tablet_service.cc:1467] Processing CreateTablet for tablet 70cd215b4f9445499e14c9fd18f7f7fe (DEFAULT_TABLE table=rebalance_test_table_3 [id=0f0d1e3fbb7e4ab888b8cadb04bf3b61]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:55.041786 13115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70cd215b4f9445499e14c9fd18f7f7fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:55.042089 12987 tablet_service.cc:1467] Processing CreateTablet for tablet 1d207517fea64fe1a83a2409157ae2bd (DEFAULT_TABLE table=rebalance_test_table_3 [id=0f0d1e3fbb7e4ab888b8cadb04bf3b61]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:55.042693 12987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d207517fea64fe1a83a2409157ae2bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:55.047994 12856 tablet_service.cc:1467] Processing CreateTablet for tablet 1d207517fea64fe1a83a2409157ae2bd (DEFAULT_TABLE table=rebalance_test_table_3 [id=0f0d1e3fbb7e4ab888b8cadb04bf3b61]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:55.048183 13116 tablet_service.cc:1467] Processing CreateTablet for tablet 1d207517fea64fe1a83a2409157ae2bd (DEFAULT_TABLE table=rebalance_test_table_3 [id=0f0d1e3fbb7e4ab888b8cadb04bf3b61]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:55.048496 12856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d207517fea64fe1a83a2409157ae2bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:55.048573 13116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d207517fea64fe1a83a2409157ae2bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:55.056915 14003 tablet_bootstrap.cc:492] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231: Bootstrap starting.
I20240501 20:10:55.060791 14003 tablet_bootstrap.cc:654] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:55.062139 14004 tablet_bootstrap.cc:492] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62: Bootstrap starting.
I20240501 20:10:55.064242 14004 tablet_bootstrap.cc:654] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:55.065326 14000 tablet_bootstrap.cc:492] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c: Bootstrap starting.
I20240501 20:10:55.066673 14001 tablet_bootstrap.cc:492] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362: No bootstrap required, opened a new log
I20240501 20:10:55.066948 14001 ts_tablet_manager.cc:1367] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.009s
I20240501 20:10:55.067636 14000 tablet_bootstrap.cc:654] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:55.067888 14001 raft_consensus.cc:348] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:55.068277 14001 raft_consensus.cc:374] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:55.068447 14001 raft_consensus.cc:729] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Initialized, Role: FOLLOWER
I20240501 20:10:55.068861 14001 consensus_queue.cc:260] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:55.069828 14001 ts_tablet_manager.cc:1398] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:10:55.070169 14001 tablet_bootstrap.cc:492] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362: Bootstrap starting.
I20240501 20:10:55.072291 14001 tablet_bootstrap.cc:654] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:55.075274 14003 tablet_bootstrap.cc:492] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231: No bootstrap required, opened a new log
I20240501 20:10:55.075491 14003 ts_tablet_manager.cc:1367] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240501 20:10:55.075946 14000 tablet_bootstrap.cc:492] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c: No bootstrap required, opened a new log
I20240501 20:10:55.076056 14004 tablet_bootstrap.cc:492] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62: No bootstrap required, opened a new log
I20240501 20:10:55.076174 14000 ts_tablet_manager.cc:1367] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20240501 20:10:55.076261 14004 ts_tablet_manager.cc:1367] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.004s
I20240501 20:10:55.076368 14003 raft_consensus.cc:348] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:55.076627 14003 raft_consensus.cc:374] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:55.076745 14003 raft_consensus.cc:729] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Initialized, Role: FOLLOWER
I20240501 20:10:55.077020 14003 consensus_queue.cc:260] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:55.077176 14004 raft_consensus.cc:348] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:55.077219 14000 raft_consensus.cc:348] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:55.077419 14004 raft_consensus.cc:374] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:55.077551 14004 raft_consensus.cc:729] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Initialized, Role: FOLLOWER
I20240501 20:10:55.077565 14000 raft_consensus.cc:374] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:55.077759 14000 raft_consensus.cc:729] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Initialized, Role: FOLLOWER
I20240501 20:10:55.077859 14004 consensus_queue.cc:260] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:55.078166 14000 consensus_queue.cc:260] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:55.079392 14004 ts_tablet_manager.cc:1398] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:10:55.079814 14004 tablet_bootstrap.cc:492] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62: Bootstrap starting.
I20240501 20:10:55.080971 14000 ts_tablet_manager.cc:1398] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:55.081286 14000 tablet_bootstrap.cc:492] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c: Bootstrap starting.
I20240501 20:10:55.081823 14001 tablet_bootstrap.cc:492] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362: No bootstrap required, opened a new log
I20240501 20:10:55.082057 14001 ts_tablet_manager.cc:1367] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.001s
I20240501 20:10:55.082967 14001 raft_consensus.cc:348] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:55.083262 14001 raft_consensus.cc:374] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:55.083390 14001 raft_consensus.cc:729] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Initialized, Role: FOLLOWER
I20240501 20:10:55.083899 14004 tablet_bootstrap.cc:654] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:55.083868 14001 consensus_queue.cc:260] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:55.084736 14001 ts_tablet_manager.cc:1398] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:10:55.085255 14001 tablet_bootstrap.cc:492] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362: Bootstrap starting.
I20240501 20:10:55.086907 14001 tablet_bootstrap.cc:654] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:55.088430 14003 ts_tablet_manager.cc:1398] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20240501 20:10:55.088786 14003 tablet_bootstrap.cc:492] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231: Bootstrap starting.
I20240501 20:10:55.090994 14003 tablet_bootstrap.cc:654] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:55.092026 14001 tablet_bootstrap.cc:492] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362: No bootstrap required, opened a new log
I20240501 20:10:55.092232 14001 ts_tablet_manager.cc:1367] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20240501 20:10:55.094287 14000 tablet_bootstrap.cc:654] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:55.094688 14001 raft_consensus.cc:348] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:55.094971 14001 raft_consensus.cc:374] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:55.095126 14001 raft_consensus.cc:729] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Initialized, Role: FOLLOWER
I20240501 20:10:55.095494 14001 consensus_queue.cc:260] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:55.096349 14001 ts_tablet_manager.cc:1398] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:55.098754 14000 tablet_bootstrap.cc:492] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c: No bootstrap required, opened a new log
I20240501 20:10:55.098943 14000 ts_tablet_manager.cc:1367] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240501 20:10:55.099169 14004 tablet_bootstrap.cc:492] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62: No bootstrap required, opened a new log
I20240501 20:10:55.099257 14003 tablet_bootstrap.cc:492] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231: No bootstrap required, opened a new log
I20240501 20:10:55.099401 14004 ts_tablet_manager.cc:1367] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.003s
I20240501 20:10:55.099462 14003 ts_tablet_manager.cc:1367] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240501 20:10:55.100095 14004 raft_consensus.cc:348] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:55.100351 14004 raft_consensus.cc:374] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:55.100255 14003 raft_consensus.cc:348] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:55.100452 14004 raft_consensus.cc:729] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Initialized, Role: FOLLOWER
I20240501 20:10:55.100522 14003 raft_consensus.cc:374] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:55.100605 14003 raft_consensus.cc:729] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Initialized, Role: FOLLOWER
I20240501 20:10:55.100767 14004 consensus_queue.cc:260] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:55.100788 14000 raft_consensus.cc:348] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:55.100855 14003 consensus_queue.cc:260] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:55.101008 14000 raft_consensus.cc:374] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:55.101117 14000 raft_consensus.cc:729] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Initialized, Role: FOLLOWER
I20240501 20:10:55.101469 14003 ts_tablet_manager.cc:1398] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:55.101398 14000 consensus_queue.cc:260] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:55.101991 14004 ts_tablet_manager.cc:1398] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:55.103087 14000 ts_tablet_manager.cc:1398] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:55.115445 13839 raft_consensus.cc:482] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:55.115649 13839 raft_consensus.cc:504] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:55.116248 13839 leader_election.cc:288] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:10:55.116804 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b8c200bb05478db26c0d10b25a514c" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9d799e45f8407590cb3096f5625c62" is_pre_election: true
I20240501 20:10:55.116791 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b8c200bb05478db26c0d10b25a514c" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c" is_pre_election: true
I20240501 20:10:55.117027 12745 raft_consensus.cc:2454] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 0.
I20240501 20:10:55.117038 12876 raft_consensus.cc:2454] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 0.
I20240501 20:10:55.117550 12942 leader_election.cc:302] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:10:55.117808 13839 raft_consensus.cc:2781] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:55.117890 13839 raft_consensus.cc:482] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:55.117967 13839 raft_consensus.cc:3037] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:55.119879 13839 raft_consensus.cc:504] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:55.120482 13839 leader_election.cc:288] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:10:55.120887 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b8c200bb05478db26c0d10b25a514c" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:10:55.120930 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b8c200bb05478db26c0d10b25a514c" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:10:55.121104 12745 raft_consensus.cc:3037] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:55.121126 12876 raft_consensus.cc:3037] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:55.123097 12876 raft_consensus.cc:2454] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 1.
I20240501 20:10:55.123224 12745 raft_consensus.cc:2454] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 1.
I20240501 20:10:55.123595 12942 leader_election.cc:302] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:10:55.123875 13839 raft_consensus.cc:2781] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:55.124281 13839 raft_consensus.cc:686] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 1 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:10:55.124521 13839 consensus_queue.cc:237] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:55.127269 12603 catalog_manager.cc:5484] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 1 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:55.190789 14009 raft_consensus.cc:482] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:55.191004 14009 raft_consensus.cc:504] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:55.192564 14009 leader_election.cc:288] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:10:55.198374 13926 raft_consensus.cc:482] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:55.198633 13926 raft_consensus.cc:504] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:55.199647 13926 leader_election.cc:288] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:10:55.200286 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70cd215b4f9445499e14c9fd18f7f7fe" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362" is_pre_election: true
I20240501 20:10:55.200620 13006 raft_consensus.cc:2454] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 0.
I20240501 20:10:55.201243 12680 leader_election.cc:302] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:10:55.201699 13926 raft_consensus.cc:2781] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:55.201836 13926 raft_consensus.cc:482] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:55.201938 13926 raft_consensus.cc:3037] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:55.205873 13926 raft_consensus.cc:504] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:55.206110 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70cd215b4f9445499e14c9fd18f7f7fe" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231" is_pre_election: true
I20240501 20:10:55.206411 13137 raft_consensus.cc:2454] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 0.
I20240501 20:10:55.207403 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70cd215b4f9445499e14c9fd18f7f7fe" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:10:55.207640 13006 raft_consensus.cc:3037] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:55.208178 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70cd215b4f9445499e14c9fd18f7f7fe" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:10:55.208375 13137 raft_consensus.cc:3037] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:55.211176 13137 raft_consensus.cc:2454] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 1.
I20240501 20:10:55.211225 13006 raft_consensus.cc:2454] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 1.
I20240501 20:10:55.211313 13926 leader_election.cc:288] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:10:55.211630 12681 leader_election.cc:302] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:10:55.212052 13926 raft_consensus.cc:2781] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:55.212822 13926 raft_consensus.cc:686] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:10:55.213196 13926 consensus_queue.cc:237] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:55.215178 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d207517fea64fe1a83a2409157ae2bd" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c" is_pre_election: true
I20240501 20:10:55.215438 12876 raft_consensus.cc:2454] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 0.
I20240501 20:10:55.215651 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d207517fea64fe1a83a2409157ae2bd" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362" is_pre_election: true
I20240501 20:10:55.215894 13006 raft_consensus.cc:2454] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 0.
I20240501 20:10:55.215970 13072 leader_election.cc:302] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:10:55.216415 14009 raft_consensus.cc:2781] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:55.216544 14009 raft_consensus.cc:482] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:55.216643 14009 raft_consensus.cc:3037] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:55.216583 12603 catalog_manager.cc:5484] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:55.218871 14009 raft_consensus.cc:504] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:55.219552 14009 leader_election.cc:288] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 1 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:10:55.219996 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d207517fea64fe1a83a2409157ae2bd" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:10:55.220117 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d207517fea64fe1a83a2409157ae2bd" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:10:55.220264 12876 raft_consensus.cc:3037] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:55.220316 13006 raft_consensus.cc:3037] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:55.223173 12876 raft_consensus.cc:2454] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 1.
I20240501 20:10:55.223189 13006 raft_consensus.cc:2454] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 1.
I20240501 20:10:55.223600 13071 leader_election.cc:302] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:10:55.223865 14009 raft_consensus.cc:2781] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:55.224222 14009 raft_consensus.cc:686] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 1 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:10:55.224632 14009 consensus_queue.cc:237] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:55.228106 12603 catalog_manager.cc:5484] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 1 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:55.512104 13839 consensus_queue.cc:1035] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:55.525907 14028 consensus_queue.cc:1035] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:55.593593 13926 consensus_queue.cc:1035] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:55.600526 13926 consensus_queue.cc:1035] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:10:55.612015 14024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:55.612350 14024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:55.619931 14024 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:10:55.645494 14009 consensus_queue.cc:1035] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:55.650696 14022 consensus_queue.cc:1035] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 267.866 ms
time per row: 0.133933 ms
I20240501 20:10:56.049485 12602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52990:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:56.058209 12726 tablet_service.cc:1467] Processing CreateTablet for tablet a04bef0a392e42ebae58a89d46e2f373 (DEFAULT_TABLE table=rebalance_test_table_4 [id=61cf4b5290874ea7958daccaf4448025]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:56.058524 12856 tablet_service.cc:1467] Processing CreateTablet for tablet a04bef0a392e42ebae58a89d46e2f373 (DEFAULT_TABLE table=rebalance_test_table_4 [id=61cf4b5290874ea7958daccaf4448025]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:56.058763 12726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a04bef0a392e42ebae58a89d46e2f373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:56.058794 13115 tablet_service.cc:1467] Processing CreateTablet for tablet a04bef0a392e42ebae58a89d46e2f373 (DEFAULT_TABLE table=rebalance_test_table_4 [id=61cf4b5290874ea7958daccaf4448025]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:56.058975 12856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a04bef0a392e42ebae58a89d46e2f373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:56.059185 13115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a04bef0a392e42ebae58a89d46e2f373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:56.060248 12985 tablet_service.cc:1467] Processing CreateTablet for tablet e89e2dfe0abc487593a61f9f1b58ac65 (DEFAULT_TABLE table=rebalance_test_table_4 [id=61cf4b5290874ea7958daccaf4448025]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:56.060650 12985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89e2dfe0abc487593a61f9f1b58ac65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:56.063001 12987 tablet_service.cc:1467] Processing CreateTablet for tablet 5af8328d0e9248acb93e3707ffde7f8a (DEFAULT_TABLE table=rebalance_test_table_4 [id=61cf4b5290874ea7958daccaf4448025]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:56.063370 12987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5af8328d0e9248acb93e3707ffde7f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:56.066359 12724 tablet_service.cc:1467] Processing CreateTablet for tablet e89e2dfe0abc487593a61f9f1b58ac65 (DEFAULT_TABLE table=rebalance_test_table_4 [id=61cf4b5290874ea7958daccaf4448025]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:56.066406 12855 tablet_service.cc:1467] Processing CreateTablet for tablet e89e2dfe0abc487593a61f9f1b58ac65 (DEFAULT_TABLE table=rebalance_test_table_4 [id=61cf4b5290874ea7958daccaf4448025]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:56.066751 12724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89e2dfe0abc487593a61f9f1b58ac65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:56.066912 12855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89e2dfe0abc487593a61f9f1b58ac65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:56.072083 14091 tablet_bootstrap.cc:492] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231: Bootstrap starting.
I20240501 20:10:56.072078 12725 tablet_service.cc:1467] Processing CreateTablet for tablet 5af8328d0e9248acb93e3707ffde7f8a (DEFAULT_TABLE table=rebalance_test_table_4 [id=61cf4b5290874ea7958daccaf4448025]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:56.073257 12725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5af8328d0e9248acb93e3707ffde7f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:56.075340 14091 tablet_bootstrap.cc:654] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:56.072763 13116 tablet_service.cc:1467] Processing CreateTablet for tablet 5af8328d0e9248acb93e3707ffde7f8a (DEFAULT_TABLE table=rebalance_test_table_4 [id=61cf4b5290874ea7958daccaf4448025]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:56.079594 13116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5af8328d0e9248acb93e3707ffde7f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:56.080300 14092 tablet_bootstrap.cc:492] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362: Bootstrap starting.
I20240501 20:10:56.081580 14090 tablet_bootstrap.cc:492] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c: Bootstrap starting.
I20240501 20:10:56.082134 14092 tablet_bootstrap.cc:654] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:56.083930 14090 tablet_bootstrap.cc:654] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:56.084311 14089 tablet_bootstrap.cc:492] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62: Bootstrap starting.
I20240501 20:10:56.086172 14089 tablet_bootstrap.cc:654] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:56.087584 14092 tablet_bootstrap.cc:492] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362: No bootstrap required, opened a new log
I20240501 20:10:56.087803 14092 ts_tablet_manager.cc:1367] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 20:10:56.088665 14092 raft_consensus.cc:348] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:56.089002 14092 raft_consensus.cc:374] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:56.089135 14092 raft_consensus.cc:729] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Initialized, Role: FOLLOWER
I20240501 20:10:56.089570 14092 consensus_queue.cc:260] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:56.090670 14092 ts_tablet_manager.cc:1398] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:56.091156 14092 tablet_bootstrap.cc:492] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362: Bootstrap starting.
I20240501 20:10:56.092784 14092 tablet_bootstrap.cc:654] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:56.096619 14092 tablet_bootstrap.cc:492] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362: No bootstrap required, opened a new log
I20240501 20:10:56.096802 14092 ts_tablet_manager.cc:1367] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240501 20:10:56.097491 14090 tablet_bootstrap.cc:492] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c: No bootstrap required, opened a new log
I20240501 20:10:56.097472 14092 raft_consensus.cc:348] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:56.097712 14090 ts_tablet_manager.cc:1367] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.014s
I20240501 20:10:56.097821 14092 raft_consensus.cc:374] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:56.097968 14092 raft_consensus.cc:729] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Initialized, Role: FOLLOWER
I20240501 20:10:56.098322 14092 consensus_queue.cc:260] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:56.098865 14090 raft_consensus.cc:348] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:56.099071 14092 ts_tablet_manager.cc:1398] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:10:56.099224 14090 raft_consensus.cc:374] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:56.099397 14090 raft_consensus.cc:729] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Initialized, Role: FOLLOWER
I20240501 20:10:56.099598 14091 tablet_bootstrap.cc:492] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231: No bootstrap required, opened a new log
I20240501 20:10:56.099812 14091 ts_tablet_manager.cc:1367] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.009s
I20240501 20:10:56.099768 14090 consensus_queue.cc:260] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:56.100646 14090 ts_tablet_manager.cc:1398] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:56.100709 14091 raft_consensus.cc:348] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:56.100967 14091 raft_consensus.cc:374] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:56.101119 14091 raft_consensus.cc:729] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Initialized, Role: FOLLOWER
I20240501 20:10:56.101459 14091 consensus_queue.cc:260] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:56.102123 14091 ts_tablet_manager.cc:1398] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:56.102651 14091 tablet_bootstrap.cc:492] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231: Bootstrap starting.
I20240501 20:10:56.103014 14090 tablet_bootstrap.cc:492] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c: Bootstrap starting.
I20240501 20:10:56.104125 14091 tablet_bootstrap.cc:654] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:56.104972 14089 tablet_bootstrap.cc:492] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62: No bootstrap required, opened a new log
I20240501 20:10:56.105039 14090 tablet_bootstrap.cc:654] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:56.105208 14089 ts_tablet_manager.cc:1367] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.005s
I20240501 20:10:56.106125 14089 raft_consensus.cc:348] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:56.106346 14089 raft_consensus.cc:374] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:56.108026 14089 raft_consensus.cc:729] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Initialized, Role: FOLLOWER
I20240501 20:10:56.108505 14089 consensus_queue.cc:260] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:56.113026 14089 ts_tablet_manager.cc:1398] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240501 20:10:56.113395 14089 tablet_bootstrap.cc:492] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62: Bootstrap starting.
I20240501 20:10:56.113731 14090 tablet_bootstrap.cc:492] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c: No bootstrap required, opened a new log
I20240501 20:10:56.113958 14090 ts_tablet_manager.cc:1367] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.008s
I20240501 20:10:56.114905 14090 raft_consensus.cc:348] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:56.115219 14090 raft_consensus.cc:374] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:56.115253 14089 tablet_bootstrap.cc:654] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:56.115418 14090 raft_consensus.cc:729] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Initialized, Role: FOLLOWER
I20240501 20:10:56.115779 14090 consensus_queue.cc:260] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:56.116426 14090 ts_tablet_manager.cc:1398] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:56.121273 14089 tablet_bootstrap.cc:492] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62: No bootstrap required, opened a new log
I20240501 20:10:56.121431 14089 ts_tablet_manager.cc:1367] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240501 20:10:56.122126 14091 tablet_bootstrap.cc:492] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231: No bootstrap required, opened a new log
I20240501 20:10:56.122047 14089 raft_consensus.cc:348] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:56.122227 14089 raft_consensus.cc:374] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:56.122280 14091 ts_tablet_manager.cc:1367] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.014s
I20240501 20:10:56.122303 14089 raft_consensus.cc:729] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Initialized, Role: FOLLOWER
I20240501 20:10:56.122550 14089 consensus_queue.cc:260] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:56.123075 14089 ts_tablet_manager.cc:1398] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:10:56.122977 14091 raft_consensus.cc:348] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:56.123194 14091 raft_consensus.cc:374] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:56.123314 14091 raft_consensus.cc:729] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Initialized, Role: FOLLOWER
I20240501 20:10:56.123438 14089 tablet_bootstrap.cc:492] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62: Bootstrap starting.
I20240501 20:10:56.123767 14091 consensus_queue.cc:260] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:56.124394 14091 ts_tablet_manager.cc:1398] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:56.124997 14089 tablet_bootstrap.cc:654] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:56.127900 14089 tablet_bootstrap.cc:492] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62: No bootstrap required, opened a new log
I20240501 20:10:56.128051 14089 ts_tablet_manager.cc:1367] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 20:10:56.128785 14089 raft_consensus.cc:348] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:56.128935 14089 raft_consensus.cc:374] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:56.129019 14089 raft_consensus.cc:729] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Initialized, Role: FOLLOWER
I20240501 20:10:56.129209 14089 consensus_queue.cc:260] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:56.129683 14089 ts_tablet_manager.cc:1398] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:10:56.138708 14069 raft_consensus.cc:482] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:56.138890 14069 raft_consensus.cc:504] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:56.139536 14069 leader_election.cc:288] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:10:56.139967 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04bef0a392e42ebae58a89d46e2f373" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9d799e45f8407590cb3096f5625c62" is_pre_election: true
I20240501 20:10:56.140056 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04bef0a392e42ebae58a89d46e2f373" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231" is_pre_election: true
I20240501 20:10:56.140203 12745 raft_consensus.cc:2454] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 0.
I20240501 20:10:56.140266 13137 raft_consensus.cc:2454] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 0.
I20240501 20:10:56.140610 12812 leader_election.cc:302] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters:
I20240501 20:10:56.140864 14069 raft_consensus.cc:2781] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:56.140970 14069 raft_consensus.cc:482] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:56.141058 14069 raft_consensus.cc:3037] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:56.143079 14069 raft_consensus.cc:504] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:56.143612 14069 leader_election.cc:288] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:10:56.144054 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04bef0a392e42ebae58a89d46e2f373" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:10:56.144116 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04bef0a392e42ebae58a89d46e2f373" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:10:56.144273 12745 raft_consensus.cc:3037] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:56.144311 13137 raft_consensus.cc:3037] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:56.146288 12745 raft_consensus.cc:2454] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 1.
I20240501 20:10:56.146294 13137 raft_consensus.cc:2454] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 1.
I20240501 20:10:56.146692 13839 raft_consensus.cc:482] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:56.146713 12812 leader_election.cc:302] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:10:56.146982 14069 raft_consensus.cc:2781] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:56.146872 13839 raft_consensus.cc:504] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:56.147171 14069 raft_consensus.cc:686] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:10:56.147476 14069 consensus_queue.cc:237] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:56.147627 13839 leader_election.cc:288] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:10:56.148097 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89e2dfe0abc487593a61f9f1b58ac65" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c" is_pre_election: true
I20240501 20:10:56.148178 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89e2dfe0abc487593a61f9f1b58ac65" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9d799e45f8407590cb3096f5625c62" is_pre_election: true
I20240501 20:10:56.148308 12876 raft_consensus.cc:2454] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 0.
I20240501 20:10:56.148411 12745 raft_consensus.cc:2454] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 0.
I20240501 20:10:56.148921 12942 leader_election.cc:302] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:10:56.149226 13839 raft_consensus.cc:2781] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:56.149317 13839 raft_consensus.cc:482] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:56.149425 13839 raft_consensus.cc:3037] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:56.151065 12602 catalog_manager.cc:5484] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 1 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:56.151393 13839 raft_consensus.cc:504] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:56.151991 13839 leader_election.cc:288] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:10:56.153038 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89e2dfe0abc487593a61f9f1b58ac65" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:10:56.153259 12745 raft_consensus.cc:3037] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:56.153254 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89e2dfe0abc487593a61f9f1b58ac65" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:10:56.153447 12876 raft_consensus.cc:3037] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:56.155297 12745 raft_consensus.cc:2454] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 1.
I20240501 20:10:56.155665 12942 leader_election.cc:302] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:10:56.155819 12876 raft_consensus.cc:2454] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 1.
I20240501 20:10:56.155917 13839 raft_consensus.cc:2781] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:56.156092 13839 raft_consensus.cc:686] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [term 1 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:10:56.156350 13839 consensus_queue.cc:237] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:56.159070 12602 catalog_manager.cc:5484] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 1 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:56.249290 14080 raft_consensus.cc:482] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:56.249460 14080 raft_consensus.cc:504] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:56.250300 14080 leader_election.cc:288] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:10:56.250788 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af8328d0e9248acb93e3707ffde7f8a" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362" is_pre_election: true
I20240501 20:10:56.251039 13007 raft_consensus.cc:2454] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 0.
I20240501 20:10:56.251499 13071 leader_election.cc:302] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:10:56.251824 14080 raft_consensus.cc:2781] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:56.251966 14080 raft_consensus.cc:482] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:56.252065 14080 raft_consensus.cc:3037] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:56.254035 14080 raft_consensus.cc:504] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:56.254695 14080 leader_election.cc:288] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 1 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:10:56.255090 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af8328d0e9248acb93e3707ffde7f8a" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:10:56.255335 13007 raft_consensus.cc:3037] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:56.256573 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af8328d0e9248acb93e3707ffde7f8a" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9d799e45f8407590cb3096f5625c62" is_pre_election: true
I20240501 20:10:56.256595 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af8328d0e9248acb93e3707ffde7f8a" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:10:56.256867 12745 raft_consensus.cc:2454] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 0.
I20240501 20:10:56.257561 13007 raft_consensus.cc:2454] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 1.
I20240501 20:10:56.257923 13071 leader_election.cc:302] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters: 4e9d799e45f8407590cb3096f5625c62
I20240501 20:10:56.258214 14080 raft_consensus.cc:2781] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:56.258345 14080 raft_consensus.cc:686] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [term 1 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:10:56.258589 14080 consensus_queue.cc:237] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:56.261485 12602 catalog_manager.cc:5484] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 1 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:56.540105 13839 consensus_queue.cc:1035] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:56.546029 13839 consensus_queue.cc:1035] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:10:56.632014 14106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:56.632347 14106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:56.639937 14106 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:10:56.667627 14069 consensus_queue.cc:1035] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:56.669785 12745 raft_consensus.cc:3037] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Advancing to term 1
Using table 'rebalance_test_table_4'
I20240501 20:10:56.673978 14022 consensus_queue.cc:1035] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:56.677115 14069 consensus_queue.cc:1035] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:56.680044 14080 consensus_queue.cc:1035] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 274.437 ms
time per row: 0.137218 ms
I20240501 20:10:57.091320 12603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52990:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:57.101105 13116 tablet_service.cc:1467] Processing CreateTablet for tablet b96f18923f9d41878cb01202e77be033 (DEFAULT_TABLE table=rebalance_test_table_5 [id=746235910235476f8c842e4977048062]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:57.101397 12987 tablet_service.cc:1467] Processing CreateTablet for tablet b96f18923f9d41878cb01202e77be033 (DEFAULT_TABLE table=rebalance_test_table_5 [id=746235910235476f8c842e4977048062]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:57.101614 13116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b96f18923f9d41878cb01202e77be033. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:57.101886 12985 tablet_service.cc:1467] Processing CreateTablet for tablet 97dfa37b2ad2421b9242de5f0c227491 (DEFAULT_TABLE table=rebalance_test_table_5 [id=746235910235476f8c842e4977048062]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:57.101881 12987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b96f18923f9d41878cb01202e77be033. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:57.102283 12985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97dfa37b2ad2421b9242de5f0c227491. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:57.102258 12856 tablet_service.cc:1467] Processing CreateTablet for tablet b96f18923f9d41878cb01202e77be033 (DEFAULT_TABLE table=rebalance_test_table_5 [id=746235910235476f8c842e4977048062]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:57.102694 12856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b96f18923f9d41878cb01202e77be033. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:57.109742 13115 tablet_service.cc:1467] Processing CreateTablet for tablet 97dfa37b2ad2421b9242de5f0c227491 (DEFAULT_TABLE table=rebalance_test_table_5 [id=746235910235476f8c842e4977048062]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:57.109869 12855 tablet_service.cc:1467] Processing CreateTablet for tablet 97dfa37b2ad2421b9242de5f0c227491 (DEFAULT_TABLE table=rebalance_test_table_5 [id=746235910235476f8c842e4977048062]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:57.110245 12855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97dfa37b2ad2421b9242de5f0c227491. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:57.110245 13115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97dfa37b2ad2421b9242de5f0c227491. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:57.111023 12857 tablet_service.cc:1467] Processing CreateTablet for tablet 66c3dd1267b741d4a5001ac7ae1d57ca (DEFAULT_TABLE table=rebalance_test_table_5 [id=746235910235476f8c842e4977048062]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:57.111424 12857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66c3dd1267b741d4a5001ac7ae1d57ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:57.114199 13117 tablet_service.cc:1467] Processing CreateTablet for tablet 66c3dd1267b741d4a5001ac7ae1d57ca (DEFAULT_TABLE table=rebalance_test_table_5 [id=746235910235476f8c842e4977048062]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:57.114606 13117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66c3dd1267b741d4a5001ac7ae1d57ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:57.114569 14175 tablet_bootstrap.cc:492] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c: Bootstrap starting.
I20240501 20:10:57.115110 12726 tablet_service.cc:1467] Processing CreateTablet for tablet 66c3dd1267b741d4a5001ac7ae1d57ca (DEFAULT_TABLE table=rebalance_test_table_5 [id=746235910235476f8c842e4977048062]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:57.115312 14174 tablet_bootstrap.cc:492] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231: Bootstrap starting.
I20240501 20:10:57.115567 12726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66c3dd1267b741d4a5001ac7ae1d57ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:57.116812 14174 tablet_bootstrap.cc:654] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:57.130970 14177 tablet_bootstrap.cc:492] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362: Bootstrap starting.
I20240501 20:10:57.132140 14178 tablet_bootstrap.cc:492] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62: Bootstrap starting.
I20240501 20:10:57.133785 14177 tablet_bootstrap.cc:654] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:57.135242 14178 tablet_bootstrap.cc:654] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:57.138147 14175 tablet_bootstrap.cc:654] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:57.138171 14174 tablet_bootstrap.cc:492] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231: No bootstrap required, opened a new log
I20240501 20:10:57.138374 14174 ts_tablet_manager.cc:1367] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.009s
I20240501 20:10:57.139621 14174 raft_consensus.cc:348] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:57.139977 14174 raft_consensus.cc:374] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:57.140120 14174 raft_consensus.cc:729] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Initialized, Role: FOLLOWER
I20240501 20:10:57.140436 14174 consensus_queue.cc:260] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:57.141341 14174 ts_tablet_manager.cc:1398] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:57.141741 14174 tablet_bootstrap.cc:492] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231: Bootstrap starting.
I20240501 20:10:57.143270 14178 tablet_bootstrap.cc:492] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62: No bootstrap required, opened a new log
I20240501 20:10:57.143543 14178 ts_tablet_manager.cc:1367] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20240501 20:10:57.143642 14174 tablet_bootstrap.cc:654] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:57.145944 14178 raft_consensus.cc:348] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:57.146317 14178 raft_consensus.cc:374] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:57.146481 14178 raft_consensus.cc:729] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Initialized, Role: FOLLOWER
I20240501 20:10:57.146831 14178 consensus_queue.cc:260] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:57.147666 14178 ts_tablet_manager.cc:1398] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:10:57.147718 14177 tablet_bootstrap.cc:492] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362: No bootstrap required, opened a new log
I20240501 20:10:57.147939 14177 ts_tablet_manager.cc:1367] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.000s
I20240501 20:10:57.148907 14177 raft_consensus.cc:348] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:57.149159 14177 raft_consensus.cc:374] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:57.149269 14177 raft_consensus.cc:729] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Initialized, Role: FOLLOWER
I20240501 20:10:57.149319 14174 tablet_bootstrap.cc:492] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231: No bootstrap required, opened a new log
I20240501 20:10:57.149380 14175 tablet_bootstrap.cc:492] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c: No bootstrap required, opened a new log
I20240501 20:10:57.149467 14174 ts_tablet_manager.cc:1367] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240501 20:10:57.149514 14175 ts_tablet_manager.cc:1367] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.005s
I20240501 20:10:57.149554 14177 consensus_queue.cc:260] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:57.150246 14175 raft_consensus.cc:348] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:57.150439 14175 raft_consensus.cc:374] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:57.150523 14175 raft_consensus.cc:729] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Initialized, Role: FOLLOWER
I20240501 20:10:57.150875 14177 ts_tablet_manager.cc:1398] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:10:57.150892 14174 raft_consensus.cc:348] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:57.150965 14175 consensus_queue.cc:260] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:57.151124 14174 raft_consensus.cc:374] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:57.151185 14177 tablet_bootstrap.cc:492] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362: Bootstrap starting.
I20240501 20:10:57.151191 14174 raft_consensus.cc:729] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Initialized, Role: FOLLOWER
I20240501 20:10:57.151535 14175 ts_tablet_manager.cc:1398] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:10:57.151410 14174 consensus_queue.cc:260] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:57.152556 14175 tablet_bootstrap.cc:492] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c: Bootstrap starting.
I20240501 20:10:57.152637 14174 ts_tablet_manager.cc:1398] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:57.152897 14174 tablet_bootstrap.cc:492] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231: Bootstrap starting.
I20240501 20:10:57.154163 14175 tablet_bootstrap.cc:654] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:57.154388 14174 tablet_bootstrap.cc:654] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:57.156878 14177 tablet_bootstrap.cc:654] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:57.163497 14177 tablet_bootstrap.cc:492] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362: No bootstrap required, opened a new log
I20240501 20:10:57.163722 14177 ts_tablet_manager.cc:1367] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20240501 20:10:57.164621 14177 raft_consensus.cc:348] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:57.164887 14177 raft_consensus.cc:374] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:57.164984 14177 raft_consensus.cc:729] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Initialized, Role: FOLLOWER
I20240501 20:10:57.165339 14177 consensus_queue.cc:260] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:57.166121 14177 ts_tablet_manager.cc:1398] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:10:57.167569 14174 tablet_bootstrap.cc:492] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231: No bootstrap required, opened a new log
I20240501 20:10:57.167752 14174 ts_tablet_manager.cc:1367] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20240501 20:10:57.167989 14175 tablet_bootstrap.cc:492] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c: No bootstrap required, opened a new log
I20240501 20:10:57.168205 14175 ts_tablet_manager.cc:1367] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20240501 20:10:57.168546 14174 raft_consensus.cc:348] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:57.168717 14174 raft_consensus.cc:374] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:57.168814 14174 raft_consensus.cc:729] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Initialized, Role: FOLLOWER
I20240501 20:10:57.169064 14175 raft_consensus.cc:348] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:57.169104 14174 consensus_queue.cc:260] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:57.169301 14175 raft_consensus.cc:374] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:57.169389 14175 raft_consensus.cc:729] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Initialized, Role: FOLLOWER
I20240501 20:10:57.169695 14174 ts_tablet_manager.cc:1398] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:10:57.169633 14175 consensus_queue.cc:260] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:57.170091 14175 ts_tablet_manager.cc:1398] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:57.170437 14175 tablet_bootstrap.cc:492] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c: Bootstrap starting.
I20240501 20:10:57.172073 14175 tablet_bootstrap.cc:654] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:57.179095 14175 tablet_bootstrap.cc:492] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c: No bootstrap required, opened a new log
I20240501 20:10:57.179250 14175 ts_tablet_manager.cc:1367] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.001s
I20240501 20:10:57.179844 14175 raft_consensus.cc:348] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:57.180019 14175 raft_consensus.cc:374] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:57.180119 14175 raft_consensus.cc:729] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Initialized, Role: FOLLOWER
I20240501 20:10:57.180337 14175 consensus_queue.cc:260] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:57.180802 14175 ts_tablet_manager.cc:1398] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:10:57.226861 14124 raft_consensus.cc:482] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:57.227077 14124 raft_consensus.cc:504] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:57.228063 14124 leader_election.cc:288] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:10:57.228495 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dfa37b2ad2421b9242de5f0c227491" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231" is_pre_election: true
I20240501 20:10:57.228603 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dfa37b2ad2421b9242de5f0c227491" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362" is_pre_election: true
I20240501 20:10:57.228775 13136 raft_consensus.cc:2454] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 0.
I20240501 20:10:57.228832 13007 raft_consensus.cc:2454] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 0.
I20240501 20:10:57.229245 12812 leader_election.cc:302] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:10:57.229506 14124 raft_consensus.cc:2781] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:57.229604 14124 raft_consensus.cc:482] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:57.229693 14124 raft_consensus.cc:3037] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:57.231616 14124 raft_consensus.cc:504] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:57.232192 14124 leader_election.cc:288] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:10:57.232689 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dfa37b2ad2421b9242de5f0c227491" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:10:57.232671 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dfa37b2ad2421b9242de5f0c227491" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:10:57.232865 13007 raft_consensus.cc:3037] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:57.232896 13136 raft_consensus.cc:3037] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:57.234864 13007 raft_consensus.cc:2454] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 1.
I20240501 20:10:57.235061 13136 raft_consensus.cc:2454] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 1.
I20240501 20:10:57.235301 12811 leader_election.cc:302] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:10:57.235592 14124 raft_consensus.cc:2781] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:57.235713 14124 raft_consensus.cc:686] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:10:57.236003 14124 consensus_queue.cc:237] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:57.239126 12597 catalog_manager.cc:5484] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 1 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:57.411768 14164 raft_consensus.cc:482] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:57.411974 14164 raft_consensus.cc:504] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:57.412716 14164 leader_election.cc:288] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:10:57.413107 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96f18923f9d41878cb01202e77be033" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c" is_pre_election: true
I20240501 20:10:57.413347 12876 raft_consensus.cc:2454] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 0.
I20240501 20:10:57.413297 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96f18923f9d41878cb01202e77be033" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362" is_pre_election: true
I20240501 20:10:57.413518 13007 raft_consensus.cc:2454] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 0.
I20240501 20:10:57.413707 13072 leader_election.cc:302] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:10:57.413988 14164 raft_consensus.cc:2781] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:57.414127 14164 raft_consensus.cc:482] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:57.414211 14164 raft_consensus.cc:3037] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:57.416285 14164 raft_consensus.cc:504] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:57.416813 14164 leader_election.cc:288] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 1 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:10:57.417240 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96f18923f9d41878cb01202e77be033" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:10:57.417330 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96f18923f9d41878cb01202e77be033" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:10:57.417431 12876 raft_consensus.cc:3037] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:57.417517 13007 raft_consensus.cc:3037] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:57.419538 13007 raft_consensus.cc:2454] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 1.
I20240501 20:10:57.419907 13071 leader_election.cc:302] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:10:57.420159 14164 raft_consensus.cc:2781] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:57.420295 14164 raft_consensus.cc:686] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 1 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:10:57.420589 14164 consensus_queue.cc:237] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:57.420871 12876 raft_consensus.cc:2454] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 1.
I20240501 20:10:57.423473 12597 catalog_manager.cc:5484] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 1 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:57.449385 14131 raft_consensus.cc:482] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:57.449609 14131 raft_consensus.cc:504] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:57.450378 14131 leader_election.cc:288] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:10:57.450942 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c3dd1267b741d4a5001ac7ae1d57ca" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231" is_pre_election: true
I20240501 20:10:57.451004 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c3dd1267b741d4a5001ac7ae1d57ca" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c" is_pre_election: true
I20240501 20:10:57.451190 13136 raft_consensus.cc:2454] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 0.
I20240501 20:10:57.451217 12876 raft_consensus.cc:2454] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 0.
I20240501 20:10:57.451718 12681 leader_election.cc:302] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters:
I20240501 20:10:57.452049 14131 raft_consensus.cc:2781] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:57.452172 14131 raft_consensus.cc:482] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:57.452265 14131 raft_consensus.cc:3037] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:57.454149 14131 raft_consensus.cc:504] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:57.454736 14131 leader_election.cc:288] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:10:57.455250 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c3dd1267b741d4a5001ac7ae1d57ca" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:10:57.455345 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c3dd1267b741d4a5001ac7ae1d57ca" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:10:57.455438 13136 raft_consensus.cc:3037] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:57.455559 12876 raft_consensus.cc:3037] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:57.457540 12876 raft_consensus.cc:2454] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 1.
I20240501 20:10:57.457984 12681 leader_election.cc:302] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters:
I20240501 20:10:57.458325 14131 raft_consensus.cc:2781] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:57.458698 14131 raft_consensus.cc:686] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:10:57.459002 14131 consensus_queue.cc:237] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:10:57.462038 12597 catalog_manager.cc:5484] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:57.481568 13136 raft_consensus.cc:2454] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 1.
I20240501 20:10:57.653066 14124 consensus_queue.cc:1035] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:57.664621 14124 consensus_queue.cc:1035] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:10:57.841468 14191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:57.841874 14191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:57.850198 14191 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:10:57.851303 14164 consensus_queue.cc:1035] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:57.861754 14080 consensus_queue.cc:1035] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:57.887530 14189 consensus_queue.cc:1035] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:57.915271 14131 consensus_queue.cc:1035] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
INSERT report
rows total: 2000
time total: 199.495 ms
time per row: 0.0997474 ms
I20240501 20:10:58.263372 12599 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52990:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:10:58.272176 12726 tablet_service.cc:1467] Processing CreateTablet for tablet b6ba30e296834fb28bf4a88141c24826 (DEFAULT_TABLE table=rebalance_test_table_6 [id=4177709722a244c2ae14bf9a1cd9b9f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:58.272334 12855 tablet_service.cc:1467] Processing CreateTablet for tablet b6ba30e296834fb28bf4a88141c24826 (DEFAULT_TABLE table=rebalance_test_table_6 [id=4177709722a244c2ae14bf9a1cd9b9f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:58.272658 12726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ba30e296834fb28bf4a88141c24826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:58.272550 12985 tablet_service.cc:1467] Processing CreateTablet for tablet b6ba30e296834fb28bf4a88141c24826 (DEFAULT_TABLE table=rebalance_test_table_6 [id=4177709722a244c2ae14bf9a1cd9b9f0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:58.272723 12855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ba30e296834fb28bf4a88141c24826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:58.272989 12985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ba30e296834fb28bf4a88141c24826. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:58.273876 12986 tablet_service.cc:1467] Processing CreateTablet for tablet 4488bf7917574118915a9355219528ee (DEFAULT_TABLE table=rebalance_test_table_6 [id=4177709722a244c2ae14bf9a1cd9b9f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:58.274317 12986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4488bf7917574118915a9355219528ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:58.279115 12987 tablet_service.cc:1467] Processing CreateTablet for tablet e1dbb57f13d348cc996b5248b7623b8e (DEFAULT_TABLE table=rebalance_test_table_6 [id=4177709722a244c2ae14bf9a1cd9b9f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:58.279510 12987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1dbb57f13d348cc996b5248b7623b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:58.279758 13117 tablet_service.cc:1467] Processing CreateTablet for tablet e1dbb57f13d348cc996b5248b7623b8e (DEFAULT_TABLE table=rebalance_test_table_6 [id=4177709722a244c2ae14bf9a1cd9b9f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:58.280167 13117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1dbb57f13d348cc996b5248b7623b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:58.280807 12857 tablet_service.cc:1467] Processing CreateTablet for tablet 4488bf7917574118915a9355219528ee (DEFAULT_TABLE table=rebalance_test_table_6 [id=4177709722a244c2ae14bf9a1cd9b9f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:58.281280 12857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4488bf7917574118915a9355219528ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:58.283562 12725 tablet_service.cc:1467] Processing CreateTablet for tablet e1dbb57f13d348cc996b5248b7623b8e (DEFAULT_TABLE table=rebalance_test_table_6 [id=4177709722a244c2ae14bf9a1cd9b9f0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:58.284121 12725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1dbb57f13d348cc996b5248b7623b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:58.290696 12724 tablet_service.cc:1467] Processing CreateTablet for tablet 4488bf7917574118915a9355219528ee (DEFAULT_TABLE table=rebalance_test_table_6 [id=4177709722a244c2ae14bf9a1cd9b9f0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:10:58.291215 12724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4488bf7917574118915a9355219528ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:10:58.291291 14258 tablet_bootstrap.cc:492] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231: Bootstrap starting.
I20240501 20:10:58.291481 14257 tablet_bootstrap.cc:492] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62: Bootstrap starting.
I20240501 20:10:58.293108 14258 tablet_bootstrap.cc:654] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:58.293881 14257 tablet_bootstrap.cc:654] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:58.300758 14258 tablet_bootstrap.cc:492] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231: No bootstrap required, opened a new log
I20240501 20:10:58.300948 14258 ts_tablet_manager.cc:1367] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20240501 20:10:58.301786 14258 raft_consensus.cc:348] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:58.302069 14258 raft_consensus.cc:374] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:58.302181 14258 raft_consensus.cc:729] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Initialized, Role: FOLLOWER
I20240501 20:10:58.302510 14257 tablet_bootstrap.cc:492] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62: No bootstrap required, opened a new log
I20240501 20:10:58.302764 14257 ts_tablet_manager.cc:1367] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20240501 20:10:58.302954 14256 tablet_bootstrap.cc:492] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362: Bootstrap starting.
I20240501 20:10:58.303395 14261 tablet_bootstrap.cc:492] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c: Bootstrap starting.
I20240501 20:10:58.303769 14258 consensus_queue.cc:260] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:58.303818 14257 raft_consensus.cc:348] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:58.304175 14257 raft_consensus.cc:374] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:58.304332 14257 raft_consensus.cc:729] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Initialized, Role: FOLLOWER
I20240501 20:10:58.304647 14257 consensus_queue.cc:260] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:58.305220 14261 tablet_bootstrap.cc:654] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:58.305655 14256 tablet_bootstrap.cc:654] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:58.307596 14258 ts_tablet_manager.cc:1398] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240501 20:10:58.312323 14261 tablet_bootstrap.cc:492] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c: No bootstrap required, opened a new log
I20240501 20:10:58.312825 14257 ts_tablet_manager.cc:1398] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240501 20:10:58.313107 14257 tablet_bootstrap.cc:492] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62: Bootstrap starting.
I20240501 20:10:58.313382 14256 tablet_bootstrap.cc:492] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362: No bootstrap required, opened a new log
I20240501 20:10:58.313366 14261 ts_tablet_manager.cc:1367] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240501 20:10:58.313536 14256 ts_tablet_manager.cc:1367] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20240501 20:10:58.315485 14256 raft_consensus.cc:348] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:58.315652 14257 tablet_bootstrap.cc:654] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:58.315716 14256 raft_consensus.cc:374] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:58.315815 14256 raft_consensus.cc:729] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Initialized, Role: FOLLOWER
I20240501 20:10:58.316129 14256 consensus_queue.cc:260] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:58.314957 14261 raft_consensus.cc:348] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:58.317539 14261 raft_consensus.cc:374] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:58.318593 14256 ts_tablet_manager.cc:1398] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240501 20:10:58.318941 14256 tablet_bootstrap.cc:492] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362: Bootstrap starting.
I20240501 20:10:58.320009 14261 raft_consensus.cc:729] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Initialized, Role: FOLLOWER
I20240501 20:10:58.320447 14261 consensus_queue.cc:260] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:58.321640 14256 tablet_bootstrap.cc:654] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:58.323719 14261 ts_tablet_manager.cc:1398] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 20:10:58.330369 14256 tablet_bootstrap.cc:492] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362: No bootstrap required, opened a new log
I20240501 20:10:58.330564 14256 ts_tablet_manager.cc:1367] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20240501 20:10:58.330703 14261 tablet_bootstrap.cc:492] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c: Bootstrap starting.
I20240501 20:10:58.331374 14256 raft_consensus.cc:348] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:58.331630 14256 raft_consensus.cc:374] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:58.331720 14256 raft_consensus.cc:729] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Initialized, Role: FOLLOWER
I20240501 20:10:58.331847 14257 tablet_bootstrap.cc:492] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62: No bootstrap required, opened a new log
I20240501 20:10:58.332044 14257 ts_tablet_manager.cc:1367] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.012s
I20240501 20:10:58.331995 14256 consensus_queue.cc:260] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:58.332504 14261 tablet_bootstrap.cc:654] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:58.332888 14257 raft_consensus.cc:348] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:58.333139 14257 raft_consensus.cc:374] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:58.333218 14257 raft_consensus.cc:729] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Initialized, Role: FOLLOWER
I20240501 20:10:58.333493 14256 ts_tablet_manager.cc:1398] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:10:58.333493 14257 consensus_queue.cc:260] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:58.333799 14256 tablet_bootstrap.cc:492] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362: Bootstrap starting.
I20240501 20:10:58.334225 14257 ts_tablet_manager.cc:1398] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:10:58.334542 14257 tablet_bootstrap.cc:492] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62: Bootstrap starting.
I20240501 20:10:58.338266 14256 tablet_bootstrap.cc:654] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:58.340059 14257 tablet_bootstrap.cc:654] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62: Neither blocks nor log segments found. Creating new log.
I20240501 20:10:58.345114 14261 tablet_bootstrap.cc:492] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c: No bootstrap required, opened a new log
I20240501 20:10:58.345352 14261 ts_tablet_manager.cc:1367] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20240501 20:10:58.346459 14261 raft_consensus.cc:348] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:58.346704 14256 tablet_bootstrap.cc:492] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362: No bootstrap required, opened a new log
I20240501 20:10:58.346784 14261 raft_consensus.cc:374] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:58.346856 14256 ts_tablet_manager.cc:1367] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20240501 20:10:58.346871 14261 raft_consensus.cc:729] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Initialized, Role: FOLLOWER
I20240501 20:10:58.347213 14261 consensus_queue.cc:260] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:58.347694 14256 raft_consensus.cc:348] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:58.347903 14256 raft_consensus.cc:374] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:58.348002 14256 raft_consensus.cc:729] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Initialized, Role: FOLLOWER
I20240501 20:10:58.348266 14256 consensus_queue.cc:260] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:58.349180 14261 ts_tablet_manager.cc:1398] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:10:58.349749 14256 ts_tablet_manager.cc:1398] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:10:58.351018 14257 tablet_bootstrap.cc:492] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62: No bootstrap required, opened a new log
I20240501 20:10:58.351184 14257 ts_tablet_manager.cc:1367] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20240501 20:10:58.351967 14257 raft_consensus.cc:348] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:58.352190 14257 raft_consensus.cc:374] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:10:58.352288 14257 raft_consensus.cc:729] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Initialized, Role: FOLLOWER
I20240501 20:10:58.352509 14257 consensus_queue.cc:260] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:58.352981 14257 ts_tablet_manager.cc:1398] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:10:58.354914 14209 raft_consensus.cc:482] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:58.355077 14209 raft_consensus.cc:504] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:58.355691 14209 leader_election.cc:288] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:10:58.356154 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1dbb57f13d348cc996b5248b7623b8e" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362" is_pre_election: true
I20240501 20:10:58.356212 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1dbb57f13d348cc996b5248b7623b8e" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231" is_pre_election: true
I20240501 20:10:58.356353 13006 raft_consensus.cc:2454] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 0.
I20240501 20:10:58.356463 13137 raft_consensus.cc:2454] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 0.
I20240501 20:10:58.356809 12680 leader_election.cc:302] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:10:58.357062 14209 raft_consensus.cc:2781] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:58.357165 14209 raft_consensus.cc:482] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:58.357250 14209 raft_consensus.cc:3037] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:58.359244 14209 raft_consensus.cc:504] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:58.359742 14209 leader_election.cc:288] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:10:58.360090 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1dbb57f13d348cc996b5248b7623b8e" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:10:58.360196 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1dbb57f13d348cc996b5248b7623b8e" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:10:58.360313 13006 raft_consensus.cc:3037] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:58.360412 13137 raft_consensus.cc:3037] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:58.363168 13006 raft_consensus.cc:2454] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 1.
I20240501 20:10:58.363168 13137 raft_consensus.cc:2454] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 1.
I20240501 20:10:58.363550 12680 leader_election.cc:302] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:10:58.363812 14189 raft_consensus.cc:2781] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:58.364017 14189 raft_consensus.cc:686] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:10:58.364327 14189 consensus_queue.cc:237] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:10:58.367067 12600 catalog_manager.cc:5484] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:58.478132 14124 raft_consensus.cc:482] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:58.478343 14124 raft_consensus.cc:504] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:58.479166 14124 leader_election.cc:288] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:10:58.479652 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ba30e296834fb28bf4a88141c24826" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9d799e45f8407590cb3096f5625c62" is_pre_election: true
I20240501 20:10:58.479729 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ba30e296834fb28bf4a88141c24826" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362" is_pre_election: true
I20240501 20:10:58.479918 12745 raft_consensus.cc:2454] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 0.
I20240501 20:10:58.479947 13006 raft_consensus.cc:2454] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 0.
I20240501 20:10:58.480340 12812 leader_election.cc:302] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters:
I20240501 20:10:58.480634 14124 raft_consensus.cc:2781] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:58.480734 14124 raft_consensus.cc:482] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:58.480851 14124 raft_consensus.cc:3037] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:58.483098 14124 raft_consensus.cc:504] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:58.483834 14124 leader_election.cc:288] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:10:58.484299 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ba30e296834fb28bf4a88141c24826" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:10:58.484361 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ba30e296834fb28bf4a88141c24826" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:10:58.484527 13006 raft_consensus.cc:3037] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:58.484580 12745 raft_consensus.cc:3037] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:58.486588 13006 raft_consensus.cc:2454] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 1.
I20240501 20:10:58.486794 12745 raft_consensus.cc:2454] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 1.
I20240501 20:10:58.487254 12812 leader_election.cc:302] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters:
I20240501 20:10:58.487536 14124 raft_consensus.cc:2781] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:58.487699 14124 raft_consensus.cc:686] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:10:58.487972 14124 consensus_queue.cc:237] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:10:58.490841 12600 catalog_manager.cc:5484] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 1 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } }
I20240501 20:10:58.660787 14124 raft_consensus.cc:482] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:10:58.660982 14124 raft_consensus.cc:504] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:58.661557 14124 leader_election.cc:288] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:10:58.662148 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4488bf7917574118915a9355219528ee" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9d799e45f8407590cb3096f5625c62" is_pre_election: true
I20240501 20:10:58.662171 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4488bf7917574118915a9355219528ee" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362" is_pre_election: true
I20240501 20:10:58.662367 12745 raft_consensus.cc:2454] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 0.
I20240501 20:10:58.662395 13006 raft_consensus.cc:2454] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 0.
I20240501 20:10:58.662817 12812 leader_election.cc:302] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters:
I20240501 20:10:58.663123 14124 raft_consensus.cc:2781] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:10:58.663218 14124 raft_consensus.cc:482] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:10:58.663290 14124 raft_consensus.cc:3037] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:58.664966 14124 raft_consensus.cc:504] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:58.665464 14124 leader_election.cc:288] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:10:58.666039 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4488bf7917574118915a9355219528ee" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:10:58.666087 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4488bf7917574118915a9355219528ee" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:10:58.666288 13006 raft_consensus.cc:3037] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:58.666288 12745 raft_consensus.cc:3037] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:10:58.669193 12745 raft_consensus.cc:2454] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 1.
I20240501 20:10:58.669189 13006 raft_consensus.cc:2454] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 1.
I20240501 20:10:58.669683 12812 leader_election.cc:302] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:10:58.670048 14124 raft_consensus.cc:2781] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:10:58.670181 14124 raft_consensus.cc:686] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:10:58.670456 14124 consensus_queue.cc:237] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:10:58.673426 12600 catalog_manager.cc:5484] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 1 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } }
I20240501 20:10:58.743816 14209 consensus_queue.cc:1035] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:58.750614 14189 consensus_queue.cc:1035] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:58.922991 14124 consensus_queue.cc:1035] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:58.931100 14124 consensus_queue.cc:1035] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:59.103101 14124 consensus_queue.cc:1035] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:10:59.109723 14281 consensus_queue.cc:1035] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:10:59.114338 14276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:10:59.114857 14276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:10:59.123322 14276 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
W20240501 20:10:59.337042 12681 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.68:37951, user_credentials={real_user=slave}} blocked reactor thread for 61842.7us
INSERT report
rows total: 2000
time total: 212.151 ms
time per row: 0.106076 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:11:00.469002 12599 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52990:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:11:00.478375 13117 tablet_service.cc:1467] Processing CreateTablet for tablet d3eac603886e4a84877d71ccebd0dff0 (DEFAULT_TABLE table=rebalance_test_table_7 [id=df79cd03997b4e40b405fb8543d9f59f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:00.478896 13117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3eac603886e4a84877d71ccebd0dff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:00.478973 12724 tablet_service.cc:1467] Processing CreateTablet for tablet f22d4331ed6f4e86b85a9919c219ab84 (DEFAULT_TABLE table=rebalance_test_table_7 [id=df79cd03997b4e40b405fb8543d9f59f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:00.479422 12724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22d4331ed6f4e86b85a9919c219ab84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:00.479363 12725 tablet_service.cc:1467] Processing CreateTablet for tablet d3eac603886e4a84877d71ccebd0dff0 (DEFAULT_TABLE table=rebalance_test_table_7 [id=df79cd03997b4e40b405fb8543d9f59f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:00.479694 12725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3eac603886e4a84877d71ccebd0dff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:00.479966 12855 tablet_service.cc:1467] Processing CreateTablet for tablet d3eac603886e4a84877d71ccebd0dff0 (DEFAULT_TABLE table=rebalance_test_table_7 [id=df79cd03997b4e40b405fb8543d9f59f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:00.480378 12855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3eac603886e4a84877d71ccebd0dff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:00.483168 12985 tablet_service.cc:1467] Processing CreateTablet for tablet f22d4331ed6f4e86b85a9919c219ab84 (DEFAULT_TABLE table=rebalance_test_table_7 [id=df79cd03997b4e40b405fb8543d9f59f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:00.483552 12985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22d4331ed6f4e86b85a9919c219ab84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:00.488083 12986 tablet_service.cc:1467] Processing CreateTablet for tablet e6a39cb39f7c4559b97ffa749a658e7b (DEFAULT_TABLE table=rebalance_test_table_7 [id=df79cd03997b4e40b405fb8543d9f59f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:00.488466 12986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a39cb39f7c4559b97ffa749a658e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:00.488642 12857 tablet_service.cc:1467] Processing CreateTablet for tablet e6a39cb39f7c4559b97ffa749a658e7b (DEFAULT_TABLE table=rebalance_test_table_7 [id=df79cd03997b4e40b405fb8543d9f59f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:00.489037 12857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a39cb39f7c4559b97ffa749a658e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:00.491015 13116 tablet_service.cc:1467] Processing CreateTablet for tablet e6a39cb39f7c4559b97ffa749a658e7b (DEFAULT_TABLE table=rebalance_test_table_7 [id=df79cd03997b4e40b405fb8543d9f59f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:00.491407 13116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a39cb39f7c4559b97ffa749a658e7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:00.488597 13115 tablet_service.cc:1467] Processing CreateTablet for tablet f22d4331ed6f4e86b85a9919c219ab84 (DEFAULT_TABLE table=rebalance_test_table_7 [id=df79cd03997b4e40b405fb8543d9f59f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:00.493408 13115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22d4331ed6f4e86b85a9919c219ab84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:00.504565 14339 tablet_bootstrap.cc:492] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231: Bootstrap starting.
I20240501 20:11:00.504873 14341 tablet_bootstrap.cc:492] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c: Bootstrap starting.
I20240501 20:11:00.506691 14341 tablet_bootstrap.cc:654] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:00.506886 14342 tablet_bootstrap.cc:492] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362: Bootstrap starting.
I20240501 20:11:00.507550 14339 tablet_bootstrap.cc:654] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:00.508085 14340 tablet_bootstrap.cc:492] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62: Bootstrap starting.
I20240501 20:11:00.509296 14342 tablet_bootstrap.cc:654] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:00.509975 14340 tablet_bootstrap.cc:654] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:00.513916 14339 tablet_bootstrap.cc:492] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231: No bootstrap required, opened a new log
I20240501 20:11:00.514199 14339 ts_tablet_manager.cc:1367] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20240501 20:11:00.515395 14339 raft_consensus.cc:348] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:00.516067 14339 raft_consensus.cc:374] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:00.516242 14339 raft_consensus.cc:729] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Initialized, Role: FOLLOWER
I20240501 20:11:00.516569 14339 consensus_queue.cc:260] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:00.517412 14339 ts_tablet_manager.cc:1398] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 20:11:00.517720 14339 tablet_bootstrap.cc:492] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231: Bootstrap starting.
I20240501 20:11:00.517817 14342 tablet_bootstrap.cc:492] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362: No bootstrap required, opened a new log
I20240501 20:11:00.517995 14342 ts_tablet_manager.cc:1367] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240501 20:11:00.518875 14342 raft_consensus.cc:348] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:00.519133 14342 raft_consensus.cc:374] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:00.519273 14342 raft_consensus.cc:729] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Initialized, Role: FOLLOWER
I20240501 20:11:00.519573 14342 consensus_queue.cc:260] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:00.520478 14342 ts_tablet_manager.cc:1398] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:00.520784 14342 tablet_bootstrap.cc:492] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362: Bootstrap starting.
I20240501 20:11:00.522378 14342 tablet_bootstrap.cc:654] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:00.519482 14339 tablet_bootstrap.cc:654] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:00.530252 14341 tablet_bootstrap.cc:492] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c: No bootstrap required, opened a new log
I20240501 20:11:00.530506 14341 ts_tablet_manager.cc:1367] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.008s
I20240501 20:11:00.531612 14341 raft_consensus.cc:348] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:00.531905 14341 raft_consensus.cc:374] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:00.532016 14341 raft_consensus.cc:729] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Initialized, Role: FOLLOWER
I20240501 20:11:00.532330 14341 consensus_queue.cc:260] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:00.534446 14341 ts_tablet_manager.cc:1398] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 20:11:00.534826 14341 tablet_bootstrap.cc:492] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c: Bootstrap starting.
I20240501 20:11:00.537237 14341 tablet_bootstrap.cc:654] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:00.540484 14339 tablet_bootstrap.cc:492] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231: No bootstrap required, opened a new log
I20240501 20:11:00.540656 14339 ts_tablet_manager.cc:1367] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.003s
I20240501 20:11:00.540720 14340 tablet_bootstrap.cc:492] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62: No bootstrap required, opened a new log
I20240501 20:11:00.540918 14340 ts_tablet_manager.cc:1367] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.026s
I20240501 20:11:00.541544 14340 raft_consensus.cc:348] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:00.541751 14340 raft_consensus.cc:374] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:00.541875 14340 raft_consensus.cc:729] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Initialized, Role: FOLLOWER
I20240501 20:11:00.541755 14339 raft_consensus.cc:348] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:00.542150 14339 raft_consensus.cc:374] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:00.542140 14340 consensus_queue.cc:260] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:00.542258 14339 raft_consensus.cc:729] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Initialized, Role: FOLLOWER
I20240501 20:11:00.542526 14339 consensus_queue.cc:260] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:00.542711 14340 ts_tablet_manager.cc:1398] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:11:00.543067 14339 ts_tablet_manager.cc:1398] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 20:11:00.543412 14339 tablet_bootstrap.cc:492] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231: Bootstrap starting.
I20240501 20:11:00.543725 14342 tablet_bootstrap.cc:492] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362: No bootstrap required, opened a new log
I20240501 20:11:00.543923 14342 ts_tablet_manager.cc:1367] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.007s
I20240501 20:11:00.543956 14340 tablet_bootstrap.cc:492] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62: Bootstrap starting.
I20240501 20:11:00.544713 14342 raft_consensus.cc:348] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:00.544953 14342 raft_consensus.cc:374] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:00.545037 14342 raft_consensus.cc:729] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Initialized, Role: FOLLOWER
I20240501 20:11:00.545215 14339 tablet_bootstrap.cc:654] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:00.545392 14318 raft_consensus.cc:482] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:00.545305 14342 consensus_queue.cc:260] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:00.545537 14318 raft_consensus.cc:504] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:00.545699 14340 tablet_bootstrap.cc:654] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:00.545876 14342 ts_tablet_manager.cc:1398] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:11:00.546252 14318 leader_election.cc:288] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:00.547012 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22d4331ed6f4e86b85a9919c219ab84" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362" is_pre_election: true
I20240501 20:11:00.547225 13007 raft_consensus.cc:2454] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 0.
I20240501 20:11:00.547621 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22d4331ed6f4e86b85a9919c219ab84" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231" is_pre_election: true
I20240501 20:11:00.548535 12680 leader_election.cc:302] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
W20240501 20:11:00.549088 12681 leader_election.cc:341] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:11:00.550487 14341 tablet_bootstrap.cc:492] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c: No bootstrap required, opened a new log
I20240501 20:11:00.550732 14339 tablet_bootstrap.cc:492] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231: No bootstrap required, opened a new log
I20240501 20:11:00.550750 14341 ts_tablet_manager.cc:1367] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
I20240501 20:11:00.550885 14339 ts_tablet_manager.cc:1367] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20240501 20:11:00.551553 14341 raft_consensus.cc:348] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:00.551681 14339 raft_consensus.cc:348] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:00.551841 14341 raft_consensus.cc:374] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:00.551892 14339 raft_consensus.cc:374] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:00.551944 14341 raft_consensus.cc:729] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Initialized, Role: FOLLOWER
I20240501 20:11:00.552022 14339 raft_consensus.cc:729] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Initialized, Role: FOLLOWER
I20240501 20:11:00.552273 14339 consensus_queue.cc:260] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:00.552284 14341 consensus_queue.cc:260] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:00.552862 14339 ts_tablet_manager.cc:1398] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:00.553035 14341 ts_tablet_manager.cc:1398] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:00.553678 14318 raft_consensus.cc:2781] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:00.553805 14318 raft_consensus.cc:482] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:00.553898 14318 raft_consensus.cc:3037] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:00.556363 14318 raft_consensus.cc:504] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:00.556921 14318 leader_election.cc:288] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:00.557300 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22d4331ed6f4e86b85a9919c219ab84" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:00.557545 13007 raft_consensus.cc:3037] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:00.560334 13007 raft_consensus.cc:2454] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 1.
I20240501 20:11:00.560779 12680 leader_election.cc:302] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:00.561100 14352 raft_consensus.cc:2781] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:00.561298 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22d4331ed6f4e86b85a9919c219ab84" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:00.561542 13137 raft_consensus.cc:3037] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:00.561633 14352 raft_consensus.cc:686] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:00.562443 14352 consensus_queue.cc:237] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:00.562592 14340 tablet_bootstrap.cc:492] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62: No bootstrap required, opened a new log
I20240501 20:11:00.563180 14340 ts_tablet_manager.cc:1367] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.010s
I20240501 20:11:00.564076 14340 raft_consensus.cc:348] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:00.564237 13137 raft_consensus.cc:2454] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 1.
I20240501 20:11:00.564332 14340 raft_consensus.cc:374] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:00.564443 14340 raft_consensus.cc:729] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Initialized, Role: FOLLOWER
I20240501 20:11:00.564760 14340 consensus_queue.cc:260] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:00.565223 14340 ts_tablet_manager.cc:1398] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 20:11:00.565804 12603 catalog_manager.cc:5484] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:00.593820 14272 raft_consensus.cc:482] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:00.594028 14272 raft_consensus.cc:504] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:00.594763 14272 leader_election.cc:288] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:00.595165 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a39cb39f7c4559b97ffa749a658e7b" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c" is_pre_election: true
I20240501 20:11:00.595278 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a39cb39f7c4559b97ffa749a658e7b" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231" is_pre_election: true
I20240501 20:11:00.595419 12876 raft_consensus.cc:2454] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 0.
I20240501 20:11:00.595538 13137 raft_consensus.cc:2454] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 0.
I20240501 20:11:00.595831 12942 leader_election.cc:302] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:00.596078 14272 raft_consensus.cc:2781] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:00.596181 14272 raft_consensus.cc:482] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:00.596284 14272 raft_consensus.cc:3037] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:00.598315 14272 raft_consensus.cc:504] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:00.598867 14272 leader_election.cc:288] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:00.599272 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a39cb39f7c4559b97ffa749a658e7b" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:00.599421 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a39cb39f7c4559b97ffa749a658e7b" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:00.599500 12876 raft_consensus.cc:3037] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:00.599632 13137 raft_consensus.cc:3037] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:00.602366 13137 raft_consensus.cc:2454] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 1.
I20240501 20:11:00.602366 12876 raft_consensus.cc:2454] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 1.
I20240501 20:11:00.602788 12942 leader_election.cc:302] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:00.603098 14272 raft_consensus.cc:2781] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:00.603502 14272 raft_consensus.cc:686] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 1 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:00.603845 14272 consensus_queue.cc:237] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:00.606540 12599 catalog_manager.cc:5484] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 1 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:00.672650 14352 raft_consensus.cc:482] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:00.672897 14352 raft_consensus.cc:504] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:00.673527 14352 leader_election.cc:288] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:00.674027 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3eac603886e4a84877d71ccebd0dff0" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231" is_pre_election: true
I20240501 20:11:00.674104 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3eac603886e4a84877d71ccebd0dff0" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c" is_pre_election: true
I20240501 20:11:00.674263 13137 raft_consensus.cc:2454] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 0.
I20240501 20:11:00.674306 12876 raft_consensus.cc:2454] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 0.
I20240501 20:11:00.674696 12681 leader_election.cc:302] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters:
I20240501 20:11:00.674983 14352 raft_consensus.cc:2781] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:00.675072 14352 raft_consensus.cc:482] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:00.675180 14352 raft_consensus.cc:3037] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:00.677095 14352 raft_consensus.cc:504] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:00.677625 14352 leader_election.cc:288] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:00.678123 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3eac603886e4a84877d71ccebd0dff0" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:00.678180 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3eac603886e4a84877d71ccebd0dff0" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:00.678357 13137 raft_consensus.cc:3037] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:00.678372 12876 raft_consensus.cc:3037] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:00.681218 13137 raft_consensus.cc:2454] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 1.
I20240501 20:11:00.681218 12876 raft_consensus.cc:2454] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 1.
I20240501 20:11:00.681624 12681 leader_election.cc:302] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters:
I20240501 20:11:00.681881 14352 raft_consensus.cc:2781] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:00.682060 14352 raft_consensus.cc:686] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:00.682291 14352 consensus_queue.cc:237] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:00.685659 12599 catalog_manager.cc:5484] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:01.031155 14352 consensus_queue.cc:1035] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:11:01.044205 14358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:01.044798 14358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 20:11:01.052170 14352 consensus_queue.cc:1035] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:11:01.058272 14358 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:11:01.087555 14365 consensus_queue.cc:1035] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:01.093262 14352 consensus_queue.cc:1035] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:01.095702 14354 consensus_queue.cc:1035] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
I20240501 20:11:01.103798 14272 consensus_queue.cc:1035] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 243.462 ms
time per row: 0.121731 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 20:11:02.407418 12601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52990:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:11:02.416615 12724 tablet_service.cc:1467] Processing CreateTablet for tablet 17276a219da74d95919c8f427c9920c5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0950cc185374470da52fb4a08455b1f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:02.416615 12985 tablet_service.cc:1467] Processing CreateTablet for tablet 17276a219da74d95919c8f427c9920c5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0950cc185374470da52fb4a08455b1f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:02.417074 12985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17276a219da74d95919c8f427c9920c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:02.417097 12724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17276a219da74d95919c8f427c9920c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:02.417640 12725 tablet_service.cc:1467] Processing CreateTablet for tablet 9a992a1cf4484b298200677ee8b56424 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0950cc185374470da52fb4a08455b1f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:02.418051 12725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a992a1cf4484b298200677ee8b56424. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:02.421064 12726 tablet_service.cc:1467] Processing CreateTablet for tablet dafd65d56bb243c78cb4490d7da7b18a (DEFAULT_TABLE table=rebalance_test_table_8 [id=0950cc185374470da52fb4a08455b1f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:02.421031 12855 tablet_service.cc:1467] Processing CreateTablet for tablet 17276a219da74d95919c8f427c9920c5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0950cc185374470da52fb4a08455b1f5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:02.421427 12855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17276a219da74d95919c8f427c9920c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:02.421437 12726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dafd65d56bb243c78cb4490d7da7b18a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:02.427057 12857 tablet_service.cc:1467] Processing CreateTablet for tablet 9a992a1cf4484b298200677ee8b56424 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0950cc185374470da52fb4a08455b1f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:02.427456 12857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a992a1cf4484b298200677ee8b56424. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:02.430083 14424 tablet_bootstrap.cc:492] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62: Bootstrap starting.
I20240501 20:11:02.430320 14425 tablet_bootstrap.cc:492] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c: Bootstrap starting.
I20240501 20:11:02.431525 13115 tablet_service.cc:1467] Processing CreateTablet for tablet 9a992a1cf4484b298200677ee8b56424 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0950cc185374470da52fb4a08455b1f5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:02.431529 13117 tablet_service.cc:1467] Processing CreateTablet for tablet dafd65d56bb243c78cb4490d7da7b18a (DEFAULT_TABLE table=rebalance_test_table_8 [id=0950cc185374470da52fb4a08455b1f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:02.431881 14424 tablet_bootstrap.cc:654] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:02.431988 13117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dafd65d56bb243c78cb4490d7da7b18a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:02.432154 14425 tablet_bootstrap.cc:654] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:02.432989 12986 tablet_service.cc:1467] Processing CreateTablet for tablet dafd65d56bb243c78cb4490d7da7b18a (DEFAULT_TABLE table=rebalance_test_table_8 [id=0950cc185374470da52fb4a08455b1f5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:02.433571 12986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dafd65d56bb243c78cb4490d7da7b18a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:02.435215 13115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a992a1cf4484b298200677ee8b56424. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:02.444041 14427 tablet_bootstrap.cc:492] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362: Bootstrap starting.
I20240501 20:11:02.446223 14427 tablet_bootstrap.cc:654] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:02.448405 14424 tablet_bootstrap.cc:492] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62: No bootstrap required, opened a new log
I20240501 20:11:02.448640 14424 ts_tablet_manager.cc:1367] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.007s
I20240501 20:11:02.449484 14424 raft_consensus.cc:348] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:02.449777 14424 raft_consensus.cc:374] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:02.449916 14424 raft_consensus.cc:729] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Initialized, Role: FOLLOWER
I20240501 20:11:02.450179 14424 consensus_queue.cc:260] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:02.451726 14424 ts_tablet_manager.cc:1398] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:11:02.452132 14424 tablet_bootstrap.cc:492] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62: Bootstrap starting.
I20240501 20:11:02.452368 14427 tablet_bootstrap.cc:492] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362: No bootstrap required, opened a new log
I20240501 20:11:02.452586 14427 ts_tablet_manager.cc:1367] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240501 20:11:02.453423 14427 raft_consensus.cc:348] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:02.453693 14427 raft_consensus.cc:374] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:02.453719 14428 tablet_bootstrap.cc:492] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231: Bootstrap starting.
I20240501 20:11:02.453855 14427 raft_consensus.cc:729] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Initialized, Role: FOLLOWER
I20240501 20:11:02.454164 14427 consensus_queue.cc:260] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:02.454847 14427 ts_tablet_manager.cc:1398] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:02.455507 14427 tablet_bootstrap.cc:492] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362: Bootstrap starting.
I20240501 20:11:02.455668 14425 tablet_bootstrap.cc:492] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c: No bootstrap required, opened a new log
I20240501 20:11:02.455893 14425 ts_tablet_manager.cc:1367] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.005s
I20240501 20:11:02.456502 14425 raft_consensus.cc:348] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:02.456679 14425 raft_consensus.cc:374] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:02.456805 14425 raft_consensus.cc:729] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Initialized, Role: FOLLOWER
I20240501 20:11:02.457051 14425 consensus_queue.cc:260] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:02.457345 14427 tablet_bootstrap.cc:654] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:02.458053 14428 tablet_bootstrap.cc:654] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:02.458509 14425 ts_tablet_manager.cc:1398] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:11:02.458691 14424 tablet_bootstrap.cc:654] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:02.458865 14425 tablet_bootstrap.cc:492] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c: Bootstrap starting.
I20240501 20:11:02.461009 14425 tablet_bootstrap.cc:654] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:02.465802 14424 tablet_bootstrap.cc:492] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62: No bootstrap required, opened a new log
I20240501 20:11:02.466060 14424 ts_tablet_manager.cc:1367] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.001s
I20240501 20:11:02.466925 14424 raft_consensus.cc:348] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:02.467221 14424 raft_consensus.cc:374] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:02.467362 14427 tablet_bootstrap.cc:492] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362: No bootstrap required, opened a new log
I20240501 20:11:02.467344 14424 raft_consensus.cc:729] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Initialized, Role: FOLLOWER
I20240501 20:11:02.467566 14427 ts_tablet_manager.cc:1367] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20240501 20:11:02.467578 14428 tablet_bootstrap.cc:492] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231: No bootstrap required, opened a new log
I20240501 20:11:02.467754 14428 ts_tablet_manager.cc:1367] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20240501 20:11:02.467700 14424 consensus_queue.cc:260] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:02.468322 14427 raft_consensus.cc:348] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:02.468569 14427 raft_consensus.cc:374] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:02.468670 14427 raft_consensus.cc:729] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Initialized, Role: FOLLOWER
I20240501 20:11:02.468698 14428 raft_consensus.cc:348] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:02.469177 14428 raft_consensus.cc:374] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:02.469303 14428 raft_consensus.cc:729] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Initialized, Role: FOLLOWER
I20240501 20:11:02.469319 14424 ts_tablet_manager.cc:1398] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 20:11:02.469568 14425 tablet_bootstrap.cc:492] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c: No bootstrap required, opened a new log
I20240501 20:11:02.469667 14424 tablet_bootstrap.cc:492] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62: Bootstrap starting.
I20240501 20:11:02.469568 14428 consensus_queue.cc:260] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:02.469765 14425 ts_tablet_manager.cc:1367] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240501 20:11:02.470201 14428 ts_tablet_manager.cc:1398] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:02.470505 14428 tablet_bootstrap.cc:492] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231: Bootstrap starting.
I20240501 20:11:02.470499 14425 raft_consensus.cc:348] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:02.470796 14425 raft_consensus.cc:374] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:02.471186 14425 raft_consensus.cc:729] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Initialized, Role: FOLLOWER
I20240501 20:11:02.468943 14427 consensus_queue.cc:260] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:02.472309 14427 ts_tablet_manager.cc:1398] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 20:11:02.472697 14424 tablet_bootstrap.cc:654] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:02.473160 14428 tablet_bootstrap.cc:654] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:02.473451 14425 consensus_queue.cc:260] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:02.475142 14425 ts_tablet_manager.cc:1398] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 20:11:02.478799 14281 raft_consensus.cc:482] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:02.479013 14281 raft_consensus.cc:504] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:02.479887 14281 leader_election.cc:288] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:02.481153 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a992a1cf4484b298200677ee8b56424" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231" is_pre_election: true
I20240501 20:11:02.481441 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a992a1cf4484b298200677ee8b56424" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9d799e45f8407590cb3096f5625c62" is_pre_election: true
I20240501 20:11:02.481623 14424 tablet_bootstrap.cc:492] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62: No bootstrap required, opened a new log
I20240501 20:11:02.481717 12746 raft_consensus.cc:2454] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 0.
I20240501 20:11:02.481896 14424 ts_tablet_manager.cc:1367] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.002s
I20240501 20:11:02.482296 12812 leader_election.cc:302] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters:
I20240501 20:11:02.482470 14428 tablet_bootstrap.cc:492] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231: No bootstrap required, opened a new log
I20240501 20:11:02.482693 14428 ts_tablet_manager.cc:1367] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
W20240501 20:11:02.482813 12812 leader_election.cc:341] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:11:02.483006 14424 raft_consensus.cc:348] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:02.483199 14424 raft_consensus.cc:374] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:02.482836 14281 raft_consensus.cc:2781] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:02.483295 14424 raft_consensus.cc:729] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Initialized, Role: FOLLOWER
I20240501 20:11:02.483458 14281 raft_consensus.cc:482] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:02.483562 14281 raft_consensus.cc:3037] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:02.483557 14428 raft_consensus.cc:348] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:02.483637 14424 consensus_queue.cc:260] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:02.483772 14428 raft_consensus.cc:374] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:02.483839 14428 raft_consensus.cc:729] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Initialized, Role: FOLLOWER
I20240501 20:11:02.484094 14428 consensus_queue.cc:260] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:02.484619 14428 ts_tablet_manager.cc:1398] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 20:11:02.485049 14424 ts_tablet_manager.cc:1398] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:11:02.486189 14281 raft_consensus.cc:504] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:02.487006 14281 leader_election.cc:288] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:02.487772 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a992a1cf4484b298200677ee8b56424" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:02.487846 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a992a1cf4484b298200677ee8b56424" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:02.487988 12746 raft_consensus.cc:3037] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:02.488039 13136 raft_consensus.cc:3037] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:02.490835 13136 raft_consensus.cc:2454] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 1.
I20240501 20:11:02.490834 12746 raft_consensus.cc:2454] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 1.
I20240501 20:11:02.491247 12812 leader_election.cc:302] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:02.491515 14437 raft_consensus.cc:2781] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:02.491678 14437 raft_consensus.cc:686] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:02.492367 14437 consensus_queue.cc:237] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:02.495553 12599 catalog_manager.cc:5484] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 1 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:02.550299 14419 raft_consensus.cc:482] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:02.550457 14419 raft_consensus.cc:504] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:02.551034 14419 leader_election.cc:288] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:02.551477 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17276a219da74d95919c8f427c9920c5" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c" is_pre_election: true
I20240501 20:11:02.551460 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17276a219da74d95919c8f427c9920c5" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9d799e45f8407590cb3096f5625c62" is_pre_election: true
I20240501 20:11:02.551735 12876 raft_consensus.cc:2454] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 0.
I20240501 20:11:02.551740 12746 raft_consensus.cc:2454] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 0.
I20240501 20:11:02.552132 12942 leader_election.cc:302] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:02.552402 14419 raft_consensus.cc:2781] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:02.552520 14419 raft_consensus.cc:482] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:02.552610 14419 raft_consensus.cc:3037] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:02.554484 14419 raft_consensus.cc:504] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:02.555032 14419 leader_election.cc:288] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:02.555383 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17276a219da74d95919c8f427c9920c5" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:02.555613 12746 raft_consensus.cc:3037] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:02.555567 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17276a219da74d95919c8f427c9920c5" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:02.555768 12876 raft_consensus.cc:3037] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:02.557497 12746 raft_consensus.cc:2454] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 1.
I20240501 20:11:02.557684 12876 raft_consensus.cc:2454] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 1.
I20240501 20:11:02.557879 12942 leader_election.cc:302] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:02.558176 14419 raft_consensus.cc:2781] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:02.558655 14419 raft_consensus.cc:686] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 1 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:02.558995 14419 consensus_queue.cc:237] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:02.562009 12599 catalog_manager.cc:5484] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 1 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:02.811925 14365 raft_consensus.cc:482] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:02.812139 14365 raft_consensus.cc:504] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:02.812776 14365 leader_election.cc:288] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:02.813324 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dafd65d56bb243c78cb4490d7da7b18a" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362" is_pre_election: true
I20240501 20:11:02.813549 13006 raft_consensus.cc:2454] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 0.
I20240501 20:11:02.813483 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dafd65d56bb243c78cb4490d7da7b18a" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231" is_pre_election: true
I20240501 20:11:02.813752 13137 raft_consensus.cc:2454] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 0.
I20240501 20:11:02.813979 12680 leader_election.cc:302] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:02.814419 14365 raft_consensus.cc:2781] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:02.814496 14365 raft_consensus.cc:482] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:02.814592 14365 raft_consensus.cc:3037] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:02.816695 14365 raft_consensus.cc:504] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:02.817255 14365 leader_election.cc:288] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:02.817643 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dafd65d56bb243c78cb4490d7da7b18a" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:02.817741 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dafd65d56bb243c78cb4490d7da7b18a" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:02.817816 13006 raft_consensus.cc:3037] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:02.817966 13137 raft_consensus.cc:3037] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:02.819661 13006 raft_consensus.cc:2454] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 1.
I20240501 20:11:02.820006 12680 leader_election.cc:302] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:02.820243 13137 raft_consensus.cc:2454] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 1.
I20240501 20:11:02.820291 14365 raft_consensus.cc:2781] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:02.820703 14365 raft_consensus.cc:686] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:02.821003 14365 consensus_queue.cc:237] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:02.823909 12599 catalog_manager.cc:5484] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:02.940709 14419 consensus_queue.cc:1035] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:02.948139 14272 consensus_queue.cc:1035] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:03.072541 14437 consensus_queue.cc:1035] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:03.077505 14437 consensus_queue.cc:1035] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:03.240669 14365 consensus_queue.cc:1035] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:03.246507 14441 consensus_queue.cc:1035] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:11:03.254297 14443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:03.254940 14443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:03.263131 14443 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
INSERT report
rows total: 2000
time total: 261.801 ms
time per row: 0.130901 ms
I20240501 20:11:03.668684 12603 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52990:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 20:11:03.677960 12857 tablet_service.cc:1467] Processing CreateTablet for tablet eeba0c03d83a433d9a519ba9fe29b53a (DEFAULT_TABLE table=rebalance_test_table_9 [id=94b2998aebca437ca7e002d981dfe3eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:03.678478 12857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeba0c03d83a433d9a519ba9fe29b53a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:03.678375 12986 tablet_service.cc:1467] Processing CreateTablet for tablet eeba0c03d83a433d9a519ba9fe29b53a (DEFAULT_TABLE table=rebalance_test_table_9 [id=94b2998aebca437ca7e002d981dfe3eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:03.678814 12986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeba0c03d83a433d9a519ba9fe29b53a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:03.679754 12987 tablet_service.cc:1467] Processing CreateTablet for tablet c1211cfd40674438a14a49818863b3f0 (DEFAULT_TABLE table=rebalance_test_table_9 [id=94b2998aebca437ca7e002d981dfe3eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:03.680164 12987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1211cfd40674438a14a49818863b3f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:03.680986 13115 tablet_service.cc:1467] Processing CreateTablet for tablet 4e077004855949878d554a2053131790 (DEFAULT_TABLE table=rebalance_test_table_9 [id=94b2998aebca437ca7e002d981dfe3eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:03.681411 13115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e077004855949878d554a2053131790. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:03.682976 12985 tablet_service.cc:1467] Processing CreateTablet for tablet 4e077004855949878d554a2053131790 (DEFAULT_TABLE table=rebalance_test_table_9 [id=94b2998aebca437ca7e002d981dfe3eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:03.683343 12985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e077004855949878d554a2053131790. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:03.684715 13117 tablet_service.cc:1467] Processing CreateTablet for tablet eeba0c03d83a433d9a519ba9fe29b53a (DEFAULT_TABLE table=rebalance_test_table_9 [id=94b2998aebca437ca7e002d981dfe3eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:03.685083 13117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeba0c03d83a433d9a519ba9fe29b53a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:03.687461 12726 tablet_service.cc:1467] Processing CreateTablet for tablet c1211cfd40674438a14a49818863b3f0 (DEFAULT_TABLE table=rebalance_test_table_9 [id=94b2998aebca437ca7e002d981dfe3eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:03.687876 12726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1211cfd40674438a14a49818863b3f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:03.691322 12855 tablet_service.cc:1467] Processing CreateTablet for tablet 4e077004855949878d554a2053131790 (DEFAULT_TABLE table=rebalance_test_table_9 [id=94b2998aebca437ca7e002d981dfe3eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:03.691740 12855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e077004855949878d554a2053131790. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:03.692003 12856 tablet_service.cc:1467] Processing CreateTablet for tablet c1211cfd40674438a14a49818863b3f0 (DEFAULT_TABLE table=rebalance_test_table_9 [id=94b2998aebca437ca7e002d981dfe3eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 20:11:03.692523 12856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1211cfd40674438a14a49818863b3f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:11:03.706007 14516 tablet_bootstrap.cc:492] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62: Bootstrap starting.
I20240501 20:11:03.709000 14516 tablet_bootstrap.cc:654] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:03.715914 14515 tablet_bootstrap.cc:492] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362: Bootstrap starting.
I20240501 20:11:03.718945 14517 tablet_bootstrap.cc:492] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c: Bootstrap starting.
I20240501 20:11:03.719105 14515 tablet_bootstrap.cc:654] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:03.720492 14519 tablet_bootstrap.cc:492] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231: Bootstrap starting.
I20240501 20:11:03.721493 14517 tablet_bootstrap.cc:654] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:03.722936 14516 tablet_bootstrap.cc:492] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62: No bootstrap required, opened a new log
I20240501 20:11:03.722956 14519 tablet_bootstrap.cc:654] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:03.723201 14516 ts_tablet_manager.cc:1367] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.010s
I20240501 20:11:03.724100 14515 tablet_bootstrap.cc:492] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362: No bootstrap required, opened a new log
I20240501 20:11:03.724121 14516 raft_consensus.cc:348] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:03.724324 14515 ts_tablet_manager.cc:1367] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240501 20:11:03.724412 14516 raft_consensus.cc:374] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:03.724550 14516 raft_consensus.cc:729] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Initialized, Role: FOLLOWER
I20240501 20:11:03.724905 14516 consensus_queue.cc:260] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:03.725170 14515 raft_consensus.cc:348] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:03.725425 14515 raft_consensus.cc:374] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:03.725564 14515 raft_consensus.cc:729] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Initialized, Role: FOLLOWER
I20240501 20:11:03.725726 14516 ts_tablet_manager.cc:1398] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 20:11:03.725903 14515 consensus_queue.cc:260] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:03.726466 14515 ts_tablet_manager.cc:1398] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 20:11:03.726810 14515 tablet_bootstrap.cc:492] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362: Bootstrap starting.
I20240501 20:11:03.728368 14515 tablet_bootstrap.cc:654] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:03.732676 14517 tablet_bootstrap.cc:492] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c: No bootstrap required, opened a new log
I20240501 20:11:03.732849 14517 ts_tablet_manager.cc:1367] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20240501 20:11:03.733712 14517 raft_consensus.cc:348] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:03.734198 14517 raft_consensus.cc:374] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:03.734318 14517 raft_consensus.cc:729] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Initialized, Role: FOLLOWER
I20240501 20:11:03.734639 14517 consensus_queue.cc:260] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:03.735379 14517 ts_tablet_manager.cc:1398] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 20:11:03.735920 14441 raft_consensus.cc:482] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:03.736112 14441 raft_consensus.cc:504] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:03.736591 14517 tablet_bootstrap.cc:492] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c: Bootstrap starting.
I20240501 20:11:03.736660 14515 tablet_bootstrap.cc:492] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362: No bootstrap required, opened a new log
I20240501 20:11:03.736821 14515 ts_tablet_manager.cc:1367] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240501 20:11:03.736879 14441 leader_election.cc:288] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:03.736981 14519 tablet_bootstrap.cc:492] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231: No bootstrap required, opened a new log
I20240501 20:11:03.737129 14519 ts_tablet_manager.cc:1367] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.006s
I20240501 20:11:03.737542 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1211cfd40674438a14a49818863b3f0" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362" is_pre_election: true
I20240501 20:11:03.737629 14515 raft_consensus.cc:348] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:03.737857 14515 raft_consensus.cc:374] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:03.737948 14515 raft_consensus.cc:729] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Initialized, Role: FOLLOWER
I20240501 20:11:03.737896 14519 raft_consensus.cc:348] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:03.738149 14519 raft_consensus.cc:374] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:03.738215 14517 tablet_bootstrap.cc:654] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:03.738243 14519 raft_consensus.cc:729] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Initialized, Role: FOLLOWER
I20240501 20:11:03.738224 14515 consensus_queue.cc:260] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:03.738656 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1211cfd40674438a14a49818863b3f0" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c" is_pre_election: true
I20240501 20:11:03.738775 14515 ts_tablet_manager.cc:1398] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:03.738449 14519 consensus_queue.cc:260] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:03.739662 13007 raft_consensus.cc:2454] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 0.
I20240501 20:11:03.739825 14515 tablet_bootstrap.cc:492] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362: Bootstrap starting.
I20240501 20:11:03.740188 14519 ts_tablet_manager.cc:1398] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 20:11:03.740476 14519 tablet_bootstrap.cc:492] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231: Bootstrap starting.
I20240501 20:11:03.742368 14519 tablet_bootstrap.cc:654] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:03.742870 14515 tablet_bootstrap.cc:654] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:03.744128 12680 leader_election.cc:302] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:03.746704 14499 raft_consensus.cc:2781] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:03.746855 14499 raft_consensus.cc:482] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:03.747033 14499 raft_consensus.cc:3037] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:03.747902 14080 raft_consensus.cc:482] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:03.748116 14080 raft_consensus.cc:504] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
W20240501 20:11:03.748409 12681 leader_election.cc:341] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:11:03.748749 14080 leader_election.cc:288] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:03.750083 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e077004855949878d554a2053131790" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c" is_pre_election: true
W20240501 20:11:03.750768 13072 leader_election.cc:341] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:11:03.751255 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e077004855949878d554a2053131790" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362" is_pre_election: true
W20240501 20:11:03.751964 13071 leader_election.cc:341] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:11:03.752121 13071 leader_election.cc:302] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb1be87a7aee48a4b405b9d209c61231; no voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362
I20240501 20:11:03.753044 14080 raft_consensus.cc:2726] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:11:03.753979 14515 tablet_bootstrap.cc:492] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362: No bootstrap required, opened a new log
I20240501 20:11:03.754230 14515 ts_tablet_manager.cc:1367] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.002s
I20240501 20:11:03.754578 14499 raft_consensus.cc:504] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:03.754878 14517 tablet_bootstrap.cc:492] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c: No bootstrap required, opened a new log
I20240501 20:11:03.755048 14517 ts_tablet_manager.cc:1367] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.001s
I20240501 20:11:03.755154 14515 raft_consensus.cc:348] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:03.755391 14515 raft_consensus.cc:374] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:03.755549 14515 raft_consensus.cc:729] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Initialized, Role: FOLLOWER
I20240501 20:11:03.755896 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1211cfd40674438a14a49818863b3f0" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:03.755836 14515 consensus_queue.cc:260] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:03.755848 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1211cfd40674438a14a49818863b3f0" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:03.755878 14517 raft_consensus.cc:348] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:03.756062 13007 raft_consensus.cc:3037] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:03.756095 14517 raft_consensus.cc:374] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:03.756171 14517 raft_consensus.cc:729] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Initialized, Role: FOLLOWER
I20240501 20:11:03.756399 14517 consensus_queue.cc:260] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:03.756542 14515 ts_tablet_manager.cc:1398] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240501 20:11:03.756937 12681 leader_election.cc:341] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867): Illegal state: must be running to vote when last-logged opid is not known
I20240501 20:11:03.757182 14499 leader_election.cc:288] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:03.757531 14517 ts_tablet_manager.cc:1398] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 20:11:03.757740 14519 tablet_bootstrap.cc:492] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231: No bootstrap required, opened a new log
I20240501 20:11:03.758056 14519 ts_tablet_manager.cc:1367] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.001s
I20240501 20:11:03.759064 14517 tablet_bootstrap.cc:492] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c: Bootstrap starting.
I20240501 20:11:03.760605 14519 raft_consensus.cc:348] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:03.760866 14519 raft_consensus.cc:374] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:03.760984 14519 raft_consensus.cc:729] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Initialized, Role: FOLLOWER
I20240501 20:11:03.761344 14519 consensus_queue.cc:260] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:03.762413 14517 tablet_bootstrap.cc:654] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c: Neither blocks nor log segments found. Creating new log.
I20240501 20:11:03.764905 13007 raft_consensus.cc:2454] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 1.
I20240501 20:11:03.765247 14519 ts_tablet_manager.cc:1398] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240501 20:11:03.765641 12680 leader_election.cc:302] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters: 77bf705599644326b6e71b5caf32fc7c
I20240501 20:11:03.766145 14499 raft_consensus.cc:2781] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:03.766322 14499 raft_consensus.cc:686] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:03.766649 14499 consensus_queue.cc:237] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:03.767210 14517 tablet_bootstrap.cc:492] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c: No bootstrap required, opened a new log
I20240501 20:11:03.767407 14517 ts_tablet_manager.cc:1367] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240501 20:11:03.768440 14517 raft_consensus.cc:348] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:03.768815 14517 raft_consensus.cc:374] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 20:11:03.768987 14517 raft_consensus.cc:729] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Initialized, Role: FOLLOWER
I20240501 20:11:03.769381 14517 consensus_queue.cc:260] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:03.770277 14517 ts_tablet_manager.cc:1398] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 20:11:03.774205 12601 catalog_manager.cc:5484] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 1 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:03.820709 14439 raft_consensus.cc:482] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:03.820928 14439 raft_consensus.cc:504] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:03.821825 14439 leader_election.cc:288] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:03.822252 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeba0c03d83a433d9a519ba9fe29b53a" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c" is_pre_election: true
I20240501 20:11:03.822358 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeba0c03d83a433d9a519ba9fe29b53a" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231" is_pre_election: true
I20240501 20:11:03.822510 12876 raft_consensus.cc:2454] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 0.
I20240501 20:11:03.822582 13137 raft_consensus.cc:2454] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 0.
I20240501 20:11:03.822983 12942 leader_election.cc:302] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:03.823282 14439 raft_consensus.cc:2781] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:03.823413 14439 raft_consensus.cc:482] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:03.823515 14439 raft_consensus.cc:3037] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:03.825534 14439 raft_consensus.cc:504] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:03.826203 14439 leader_election.cc:288] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:03.826690 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeba0c03d83a433d9a519ba9fe29b53a" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:03.826727 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeba0c03d83a433d9a519ba9fe29b53a" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:03.826900 13137 raft_consensus.cc:3037] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:03.826939 12876 raft_consensus.cc:3037] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:03.828812 13137 raft_consensus.cc:2454] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 1.
I20240501 20:11:03.828879 12876 raft_consensus.cc:2454] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 1.
I20240501 20:11:03.829210 12942 leader_election.cc:302] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:03.829466 14439 raft_consensus.cc:2781] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:03.829602 14439 raft_consensus.cc:686] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 1 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:03.829893 14439 consensus_queue.cc:237] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:03.832931 12601 catalog_manager.cc:5484] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 1 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:03.852787 14450 raft_consensus.cc:482] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:03.852985 14450 raft_consensus.cc:504] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:03.853704 14450 leader_election.cc:288] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:03.854255 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e077004855949878d554a2053131790" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362" is_pre_election: true
I20240501 20:11:03.854298 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e077004855949878d554a2053131790" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231" is_pre_election: true
I20240501 20:11:03.854514 13137 raft_consensus.cc:2454] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 0.
I20240501 20:11:03.854511 13007 raft_consensus.cc:2454] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 0.
I20240501 20:11:03.854997 12812 leader_election.cc:302] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:03.855293 14450 raft_consensus.cc:2781] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 20:11:03.855404 14450 raft_consensus.cc:482] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 20:11:03.855495 14450 raft_consensus.cc:3037] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:03.858155 14450 raft_consensus.cc:504] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:03.859318 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e077004855949878d554a2053131790" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:03.859540 13007 raft_consensus.cc:3037] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:03.859489 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e077004855949878d554a2053131790" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:03.859689 13137 raft_consensus.cc:3037] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:03.862391 13137 raft_consensus.cc:2454] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 1.
I20240501 20:11:03.862409 13007 raft_consensus.cc:2454] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 1.
I20240501 20:11:03.862921 12812 leader_election.cc:302] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:03.863265 14503 raft_consensus.cc:2781] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 20:11:03.863407 14503 raft_consensus.cc:686] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:03.863737 14503 consensus_queue.cc:237] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:03.865396 14450 leader_election.cc:288] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:03.867585 12601 catalog_manager.cc:5484] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 0 to 1, leader changed from <none> to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 1 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:03.903995 14450 raft_consensus.cc:482] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 20:11:03.904194 14450 raft_consensus.cc:504] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:03.904863 14450 leader_election.cc:288] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:03.905360 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1211cfd40674438a14a49818863b3f0" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4720929b44c4560bfdf08107e137362" is_pre_election: true
I20240501 20:11:03.905632 13007 raft_consensus.cc:2379] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c in current term 1: Already voted for candidate 4e9d799e45f8407590cb3096f5625c62 in this term.
I20240501 20:11:03.905511 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1211cfd40674438a14a49818863b3f0" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9d799e45f8407590cb3096f5625c62" is_pre_election: true
I20240501 20:11:03.906276 12812 leader_election.cc:302] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c; no voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362
I20240501 20:11:03.906649 14450 raft_consensus.cc:3037] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 20:11:03.908704 14450 raft_consensus.cc:2726] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 20:11:04.223380 14499 consensus_queue.cc:1035] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:11:04.230127 14533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:04.230598 14533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 20:11:04.231261 14441 consensus_queue.cc:1035] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 20:11:04.240911 14533 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 20:11:04.272547 14503 consensus_queue.cc:1035] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
I20240501 20:11:04.278942 14503 consensus_queue.cc:1035] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:04.296376 14419 consensus_queue.cc:1035] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 20:11:04.301743 14439 consensus_queue.cc:1035] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 230.79 ms
time per row: 0.115395 ms
I20240501 20:11:04.630499 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.69:33377
--local_ip_for_outbound_sockets=127.29.219.69
--tserver_master_addrs=127.29.219.126:34159
--webserver_port=40979
--webserver_interface=127.29.219.69
--builtin_ntp_servers=127.29.219.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 20:11:04.992245 14592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:04.992544 14592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:04.992630 14592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:05.000386 14592 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:05.000543 14592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:05.000674 14592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.69
I20240501 20:11:05.011054 14592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.69:33377
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.29.219.69
--webserver_port=40979
--tserver_master_addrs=127.29.219.126:34159
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.69
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:05.012528 14592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:05.014251 14592 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:05.026475 14597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:05.027134 14600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:05.027670 14598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:05.027974 14592 server_base.cc:1034] running on GCE node
I20240501 20:11:05.028578 14592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:05.029561 14592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:05.030762 14592 hybrid_clock.cc:648] HybridClock initialized: now 1714594265030715 us; error 45 us; skew 500 ppm
I20240501 20:11:05.031096 14592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:05.033344 14592 webserver.cc:460] Webserver started at http://127.29.219.69:40979/ using document root <none> and password file <none>
I20240501 20:11:05.034094 14592 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:05.034220 14592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:05.039777 14592 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240501 20:11:05.042603 14606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:05.043978 14592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 20:11:05.044129 14592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "ad74bd6a74e840919906ee61ce3560a4"
format_stamp: "Formatted at 2024-05-01 20:10:45 on dist-test-slave-h2cj"
I20240501 20:11:05.044905 14592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:05.086548 14592 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:05.087568 14592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:05.087886 14592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:05.088961 14592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:05.090998 14592 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:05.091081 14592 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:05.091179 14592 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:05.091228 14592 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:05.125648 14592 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.69:33377
I20240501 20:11:05.125684 14718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.69:33377 every 8 connection(s)
I20240501 20:11:05.127146 14592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 20:11:05.136011 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 14592
I20240501 20:11:05.136628 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.70:41485
--local_ip_for_outbound_sockets=127.29.219.70
--tserver_master_addrs=127.29.219.126:34159
--webserver_port=45481
--webserver_interface=127.29.219.70
--builtin_ntp_servers=127.29.219.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:05.140692 14719 heartbeater.cc:347] Connected to a master server at 127.29.219.126:34159
I20240501 20:11:05.141021 14719 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:05.144316 14719 heartbeater.cc:511] Master 127.29.219.126:34159 requested a full tablet report, sending...
I20240501 20:11:05.146076 12599 ts_manager.cc:194] Re-registered known tserver with Master: ad74bd6a74e840919906ee61ce3560a4 (127.29.219.69:33377)
I20240501 20:11:05.146976 12599 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.69:34245
W20240501 20:11:05.492931 14723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:05.493208 14723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:05.493324 14723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:05.501005 14723 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:05.501138 14723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:05.501278 14723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.70
I20240501 20:11:05.512252 14723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.70:41485
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.29.219.70
--webserver_port=45481
--tserver_master_addrs=127.29.219.126:34159
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.70
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:05.513657 14723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:05.515789 14723 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:05.527650 14729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:05.528239 14730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:05.529021 14732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:05.529378 14723 server_base.cc:1034] running on GCE node
I20240501 20:11:05.529946 14723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:05.530864 14723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:05.532089 14723 hybrid_clock.cc:648] HybridClock initialized: now 1714594265532034 us; error 42 us; skew 500 ppm
I20240501 20:11:05.532423 14723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:05.534540 14723 webserver.cc:460] Webserver started at http://127.29.219.70:45481/ using document root <none> and password file <none>
I20240501 20:11:05.535254 14723 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:05.535356 14723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:05.540809 14723 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 20:11:05.543993 14738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:05.545469 14723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 20:11:05.545604 14723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6c07aec53e944308aa4630aee955ba38"
format_stamp: "Formatted at 2024-05-01 20:10:45 on dist-test-slave-h2cj"
I20240501 20:11:05.546568 14723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:05.568663 14723 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:05.569581 14723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:05.569851 14723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:05.570793 14723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:05.572355 14723 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:05.572433 14723 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:05.572558 14723 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:05.572615 14723 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:05.607595 14723 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.70:41485
I20240501 20:11:05.607625 14850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.70:41485 every 8 connection(s)
I20240501 20:11:05.609205 14723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 20:11:05.612998 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 14723
I20240501 20:11:05.613708 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.71:46811
--local_ip_for_outbound_sockets=127.29.219.71
--tserver_master_addrs=127.29.219.126:34159
--webserver_port=40565
--webserver_interface=127.29.219.71
--builtin_ntp_servers=127.29.219.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:05.623674 14851 heartbeater.cc:347] Connected to a master server at 127.29.219.126:34159
I20240501 20:11:05.624033 14851 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:05.625072 14851 heartbeater.cc:511] Master 127.29.219.126:34159 requested a full tablet report, sending...
I20240501 20:11:05.626664 12599 ts_manager.cc:194] Re-registered known tserver with Master: 6c07aec53e944308aa4630aee955ba38 (127.29.219.70:41485)
I20240501 20:11:05.627483 12599 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.70:41257
W20240501 20:11:05.969480 14854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:05.969779 14854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:05.969897 14854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:05.977499 14854 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:05.977641 14854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:05.977807 14854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.71
I20240501 20:11:05.987224 14854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.71:46811
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.29.219.71
--webserver_port=40565
--tserver_master_addrs=127.29.219.126:34159
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.71
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:05.988600 14854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:05.990442 14854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:06.002436 14862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:06.003350 14861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:06.003939 14864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:06.004128 14854 server_base.cc:1034] running on GCE node
I20240501 20:11:06.004747 14854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:06.006178 14854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:06.007474 14854 hybrid_clock.cc:648] HybridClock initialized: now 1714594266007418 us; error 48 us; skew 500 ppm
I20240501 20:11:06.007828 14854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:06.010107 14854 webserver.cc:460] Webserver started at http://127.29.219.71:40565/ using document root <none> and password file <none>
I20240501 20:11:06.010872 14854 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:06.011004 14854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:06.016193 14854 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20240501 20:11:06.019026 14870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:06.020253 14854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240501 20:11:06.020413 14854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "9053a862fa454257992c47064f1a075d"
format_stamp: "Formatted at 2024-05-01 20:10:46 on dist-test-slave-h2cj"
I20240501 20:11:06.021371 14854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:06.055886 14854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:06.056842 14854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:06.057133 14854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:06.058156 14854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:06.059867 14854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:06.059957 14854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:06.060099 14854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:06.060160 14854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:06.099782 14854 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.71:46811
I20240501 20:11:06.099833 14982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.71:46811 every 8 connection(s)
I20240501 20:11:06.101392 14854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 20:11:06.110950 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 14854
I20240501 20:11:06.111632 30573 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
/tmp/dist-test-taskVKrfcN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.219.72:33839
--local_ip_for_outbound_sockets=127.29.219.72
--tserver_master_addrs=127.29.219.126:34159
--webserver_port=37019
--webserver_interface=127.29.219.72
--builtin_ntp_servers=127.29.219.84:39159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 20:11:06.118160 14983 heartbeater.cc:347] Connected to a master server at 127.29.219.126:34159
I20240501 20:11:06.118494 14983 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:06.119596 14983 heartbeater.cc:511] Master 127.29.219.126:34159 requested a full tablet report, sending...
I20240501 20:11:06.121237 12599 ts_manager.cc:194] Re-registered known tserver with Master: 9053a862fa454257992c47064f1a075d (127.29.219.71:46811)
I20240501 20:11:06.122501 12599 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.71:55663
I20240501 20:11:06.149669 14719 heartbeater.cc:503] Master 127.29.219.126:34159 was elected leader, sending a full tablet report...
W20240501 20:11:06.462133 14986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 20:11:06.462448 14986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 20:11:06.462558 14986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 20:11:06.470182 14986 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 20:11:06.470299 14986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 20:11:06.470417 14986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.219.72
I20240501 20:11:06.479269 14986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.219.84:39159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.219.72:33839
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.29.219.72
--webserver_port=37019
--tserver_master_addrs=127.29.219.126:34159
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.219.72
--log_dir=/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 19:43:21 UTC on efd1337a2aa3
build id 79
ASAN enabled
I20240501 20:11:06.480506 14986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 20:11:06.482163 14986 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 20:11:06.493520 14991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:06.494210 14992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 20:11:06.495033 14994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 20:11:06.495280 14986 server_base.cc:1034] running on GCE node
I20240501 20:11:06.495827 14986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 20:11:06.496770 14986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 20:11:06.497993 14986 hybrid_clock.cc:648] HybridClock initialized: now 1714594266497921 us; error 57 us; skew 500 ppm
I20240501 20:11:06.498311 14986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 20:11:06.500711 14986 webserver.cc:460] Webserver started at http://127.29.219.72:37019/ using document root <none> and password file <none>
I20240501 20:11:06.501435 14986 fs_manager.cc:359] Metadata directory not provided
I20240501 20:11:06.501554 14986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 20:11:06.506598 14986 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 20:11:06.509342 15000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:06.510726 14986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240501 20:11:06.510903 14986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "078b7bc667e2484f9f25124462c84da9"
format_stamp: "Formatted at 2024-05-01 20:10:46 on dist-test-slave-h2cj"
I20240501 20:11:06.511888 14986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 20:11:06.539654 14986 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 20:11:06.540522 14986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 20:11:06.540762 14986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 20:11:06.541736 14986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 20:11:06.543512 14986 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 20:11:06.543586 14986 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:06.543680 14986 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 20:11:06.543721 14986 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 20:11:06.579689 14986 rpc_server.cc:307] RPC server started. Bound to: 127.29.219.72:33839
I20240501 20:11:06.579739 15113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.219.72:33839 every 8 connection(s)
I20240501 20:11:06.581146 14986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVKrfcN/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714594112840080-30573-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 20:11:06.589061 30573 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVKrfcN/build/asan/bin/kudu as pid 14986
I20240501 20:11:06.595695 15114 heartbeater.cc:347] Connected to a master server at 127.29.219.126:34159
I20240501 20:11:06.596060 15114 heartbeater.cc:464] Registering TS with master...
I20240501 20:11:06.597163 15114 heartbeater.cc:511] Master 127.29.219.126:34159 requested a full tablet report, sending...
I20240501 20:11:06.599030 12600 ts_manager.cc:194] Re-registered known tserver with Master: 078b7bc667e2484f9f25124462c84da9 (127.29.219.72:33839)
I20240501 20:11:06.600452 12600 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.219.72:40635
I20240501 20:11:06.630652 14851 heartbeater.cc:503] Master 127.29.219.126:34159 was elected leader, sending a full tablet report...
I20240501 20:11:06.643146 13136 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eeba0c03d83a433d9a519ba9fe29b53a"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:06.643421 13136 raft_consensus.cc:482] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:06.643543 13136 raft_consensus.cc:3037] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.645888 13136 raft_consensus.cc:504] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:06.646802 13136 leader_election.cc:288] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:06.647503 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeba0c03d83a433d9a519ba9fe29b53a" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:06.647488 12877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeba0c03d83a433d9a519ba9fe29b53a" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:06.647696 13007 raft_consensus.cc:3032] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:06.647754 12877 raft_consensus.cc:3037] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.647794 13007 raft_consensus.cc:729] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:06.648072 13007 consensus_queue.cc:260] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:06.648545 13007 raft_consensus.cc:3037] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.650749 13007 raft_consensus.cc:2454] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.651607 13071 leader_election.cc:302] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:06.652006 14527 raft_consensus.cc:2781] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:06.652730 13136 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e077004855949878d554a2053131790"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:06.652994 13136 raft_consensus.cc:482] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:06.653185 13136 raft_consensus.cc:3037] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.654879 12877 raft_consensus.cc:2454] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.655275 14527 raft_consensus.cc:686] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:06.655474 13136 raft_consensus.cc:504] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:06.655660 14527 consensus_queue.cc:237] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:06.656114 13136 leader_election.cc:288] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:06.656603 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e077004855949878d554a2053131790" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:06.656769 13007 raft_consensus.cc:3037] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.658087 12877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e077004855949878d554a2053131790" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:06.658298 12877 raft_consensus.cc:3032] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:06.658404 12877 raft_consensus.cc:729] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:06.659444 13007 raft_consensus.cc:2454] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.659723 13136 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e6a39cb39f7c4559b97ffa749a658e7b"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:06.659915 13071 leader_election.cc:302] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:06.659936 13136 raft_consensus.cc:482] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:06.660094 13136 raft_consensus.cc:3037] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.659576 12877 consensus_queue.cc:260] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:06.660877 15123 raft_consensus.cc:2781] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:06.663218 13136 raft_consensus.cc:504] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:06.663334 12600 catalog_manager.cc:5484] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 1 to 2, leader changed from b4720929b44c4560bfdf08107e137362 (127.29.219.67) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 2 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:06.664623 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a39cb39f7c4559b97ffa749a658e7b" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:06.664884 12876 raft_consensus.cc:3037] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.666373 12877 raft_consensus.cc:3037] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.668689 12876 raft_consensus.cc:2454] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.669317 13072 leader_election.cc:302] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:06.669759 14527 raft_consensus.cc:2781] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:06.669950 14527 raft_consensus.cc:686] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:06.670298 14527 consensus_queue.cc:237] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:06.670514 12877 raft_consensus.cc:2454] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.674211 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a39cb39f7c4559b97ffa749a658e7b" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:06.674507 13007 raft_consensus.cc:3032] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:06.674602 13007 raft_consensus.cc:729] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:06.675005 13007 consensus_queue.cc:260] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:06.675453 13007 raft_consensus.cc:3037] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.679260 13007 raft_consensus.cc:2454] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.680538 13136 leader_election.cc:288] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:06.685423 15123 raft_consensus.cc:686] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:06.685958 15123 consensus_queue.cc:237] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:06.691910 12600 catalog_manager.cc:5484] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 1 to 2, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 2 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:06.693969 13136 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dafd65d56bb243c78cb4490d7da7b18a"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:06.694248 13136 raft_consensus.cc:482] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:06.694413 13136 raft_consensus.cc:3037] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.697533 13136 raft_consensus.cc:504] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:06.698722 13136 leader_election.cc:288] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:06.698765 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dafd65d56bb243c78cb4490d7da7b18a" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:06.698859 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dafd65d56bb243c78cb4490d7da7b18a" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:06.699065 13007 raft_consensus.cc:3037] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.699064 12745 raft_consensus.cc:3032] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:06.699148 12745 raft_consensus.cc:729] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:06.699496 12745 consensus_queue.cc:260] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:06.700013 12745 raft_consensus.cc:3037] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.701874 13007 raft_consensus.cc:2454] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.702428 13071 leader_election.cc:302] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:06.702754 12745 raft_consensus.cc:2454] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.702601 12599 catalog_manager.cc:5484] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 1 to 2, leader changed from b4720929b44c4560bfdf08107e137362 (127.29.219.67) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 2 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:06.704062 13136 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "248dfbfcb6214244a410083e8de22743"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:06.704233 13136 raft_consensus.cc:482] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:06.704326 13136 raft_consensus.cc:3037] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.705914 15130 raft_consensus.cc:2781] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:06.706081 15130 raft_consensus.cc:686] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:06.706429 15130 consensus_queue.cc:237] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:06.706609 13136 raft_consensus.cc:504] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:06.707849 13136 leader_election.cc:288] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:06.707841 12877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "248dfbfcb6214244a410083e8de22743" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:06.707938 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "248dfbfcb6214244a410083e8de22743" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:06.708076 12877 raft_consensus.cc:3032] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:06.708173 12877 raft_consensus.cc:729] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:06.708160 12745 raft_consensus.cc:3037] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.708421 12877 consensus_queue.cc:260] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:06.708768 12877 raft_consensus.cc:3037] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.708868 13136 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a8b101e9e38b4048bc86d76f97201534"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:06.709038 13136 raft_consensus.cc:482] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:06.709141 13136 raft_consensus.cc:3037] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.711144 12745 raft_consensus.cc:2454] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.711534 13072 leader_election.cc:302] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:06.711817 15123 raft_consensus.cc:2781] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:06.712052 15123 raft_consensus.cc:686] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:06.712323 15123 consensus_queue.cc:237] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:06.712450 13136 raft_consensus.cc:504] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:06.712930 12877 raft_consensus.cc:2454] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.713901 12877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b101e9e38b4048bc86d76f97201534" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:06.714156 12877 raft_consensus.cc:3032] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:06.714283 12877 raft_consensus.cc:729] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:06.714704 12877 consensus_queue.cc:260] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:06.715210 12877 raft_consensus.cc:3037] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.718853 12877 raft_consensus.cc:2454] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.719451 13072 leader_election.cc:302] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:06.719861 15125 raft_consensus.cc:2781] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:06.720041 15125 raft_consensus.cc:686] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:06.720435 15125 consensus_queue.cc:237] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:06.721647 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b101e9e38b4048bc86d76f97201534" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:06.721968 13007 raft_consensus.cc:3037] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.725697 13007 raft_consensus.cc:2454] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.726415 13136 leader_election.cc:288] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:06.727957 13137 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c3933feed404a4baee16428d2aafacf"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:06.728214 13137 raft_consensus.cc:482] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:06.728348 13137 raft_consensus.cc:3037] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.733438 13137 raft_consensus.cc:504] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:06.734153 13137 leader_election.cc:288] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:06.735296 13137 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "436330ea0516443799906257122b38af"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:06.735340 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c3933feed404a4baee16428d2aafacf" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:06.735493 13137 raft_consensus.cc:482] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:06.735589 13137 raft_consensus.cc:3037] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.735635 13007 raft_consensus.cc:3032] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:06.735716 13007 raft_consensus.cc:729] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:06.735986 13007 consensus_queue.cc:260] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:06.736336 13007 raft_consensus.cc:3037] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.738107 13137 raft_consensus.cc:504] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:06.738224 13007 raft_consensus.cc:2454] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.738651 12877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c3933feed404a4baee16428d2aafacf" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:06.738801 13071 leader_election.cc:302] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:06.738881 12877 raft_consensus.cc:3037] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.740393 12599 catalog_manager.cc:5484] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 1 to 2, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 2 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:06.742206 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436330ea0516443799906257122b38af" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:06.742431 12745 raft_consensus.cc:3037] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.742923 13137 leader_election.cc:288] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:06.744061 13135 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5f728b714ca142648ba5aa3459f7ae80"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:06.744253 13135 raft_consensus.cc:482] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:06.744364 13135 raft_consensus.cc:3037] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.744920 12745 raft_consensus.cc:2454] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.745397 13072 leader_election.cc:302] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:06.745707 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436330ea0516443799906257122b38af" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:06.745891 13007 raft_consensus.cc:3032] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:06.746098 13007 raft_consensus.cc:729] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:06.746371 13007 consensus_queue.cc:260] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:06.746697 13007 raft_consensus.cc:3037] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.748719 13007 raft_consensus.cc:2454] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.749140 13135 raft_consensus.cc:504] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:06.749817 13135 leader_election.cc:288] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:06.749696 12599 catalog_manager.cc:5484] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 1 to 2, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 2 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:06.750155 12877 raft_consensus.cc:2454] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.750914 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f728b714ca142648ba5aa3459f7ae80" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:06.751137 15125 raft_consensus.cc:2781] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:06.751170 12876 raft_consensus.cc:3037] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.751152 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f728b714ca142648ba5aa3459f7ae80" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:06.751291 15125 raft_consensus.cc:686] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:06.751389 12745 raft_consensus.cc:3032] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:06.751473 12745 raft_consensus.cc:729] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:06.751518 15125 consensus_queue.cc:237] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:06.751749 12745 consensus_queue.cc:260] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:06.752125 12745 raft_consensus.cc:3037] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.752591 15130 raft_consensus.cc:2781] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:06.752828 15130 raft_consensus.cc:686] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:06.753051 15130 consensus_queue.cc:237] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:06.753924 12876 raft_consensus.cc:2454] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.754037 12599 catalog_manager.cc:5484] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 1 to 2, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 2 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:06.754792 13072 leader_election.cc:302] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:06.755158 14527 raft_consensus.cc:2781] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:06.755239 13136 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f22d4331ed6f4e86b85a9919c219ab84"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:06.755321 14527 raft_consensus.cc:686] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:06.755681 14527 consensus_queue.cc:237] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:06.756556 12745 raft_consensus.cc:2454] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.758546 13136 raft_consensus.cc:482] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:06.758972 13136 raft_consensus.cc:3037] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.766268 13136 raft_consensus.cc:504] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:06.768283 13136 leader_election.cc:288] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:06.769431 13136 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "97dfa37b2ad2421b9242de5f0c227491"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:06.769681 13136 raft_consensus.cc:482] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:06.769855 13136 raft_consensus.cc:3037] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.770829 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22d4331ed6f4e86b85a9919c219ab84" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:06.771036 13006 raft_consensus.cc:3037] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.772902 13006 raft_consensus.cc:2454] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.773362 13071 leader_election.cc:302] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:06.773707 15139 raft_consensus.cc:2781] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:06.773896 15139 raft_consensus.cc:686] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:06.774335 15139 consensus_queue.cc:237] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:06.776937 13136 raft_consensus.cc:504] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:06.779003 13136 leader_election.cc:288] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:06.779202 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dfa37b2ad2421b9242de5f0c227491" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:06.779521 13006 raft_consensus.cc:3037] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.780210 13136 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e00b9b3645a04a4686e5759bd8201eca"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:06.780414 13136 raft_consensus.cc:482] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:06.780524 13136 raft_consensus.cc:3037] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.783613 13006 raft_consensus.cc:2454] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.784373 13071 leader_election.cc:302] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:06.786247 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22d4331ed6f4e86b85a9919c219ab84" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:06.786676 12745 raft_consensus.cc:3032] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:06.786862 12745 raft_consensus.cc:729] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:06.787314 12745 consensus_queue.cc:260] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:06.788650 12745 raft_consensus.cc:3037] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.789430 13136 raft_consensus.cc:504] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:06.789920 13137 raft_consensus.cc:1226] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Rejecting Update request from peer 4e9d799e45f8407590cb3096f5625c62 for earlier term 1. Current term is 2. Ops: []
I20240501 20:11:06.790107 13136 leader_election.cc:288] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:06.791070 13137 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "66c3dd1267b741d4a5001ac7ae1d57ca"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:06.791276 13137 raft_consensus.cc:482] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:06.791376 13137 raft_consensus.cc:3037] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.791400 14586 consensus_queue.cc:1046] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:06.794107 13137 raft_consensus.cc:504] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:06.794854 13137 leader_election.cc:288] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:06.795819 13135 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "50561f1e0c5b47a78abb1fba84e725fa"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:06.796079 13135 raft_consensus.cc:482] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:06.796216 13135 raft_consensus.cc:3037] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.799389 13135 raft_consensus.cc:504] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:06.800199 13135 leader_election.cc:288] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:06.801270 13134 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1d207517fea64fe1a83a2409157ae2bd"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:06.801558 13134 raft_consensus.cc:465] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 20:11:06.802834 13134 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "70cd215b4f9445499e14c9fd18f7f7fe"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:06.803053 13134 raft_consensus.cc:482] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:06.803206 13134 raft_consensus.cc:3037] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.803889 15139 raft_consensus.cc:2781] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:06.804170 15139 raft_consensus.cc:686] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:06.804457 15139 consensus_queue.cc:237] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:06.806890 13134 raft_consensus.cc:504] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:06.807932 13134 leader_election.cc:288] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:06.808527 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70cd215b4f9445499e14c9fd18f7f7fe" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:06.808781 13007 raft_consensus.cc:3037] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.811596 13136 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a04bef0a392e42ebae58a89d46e2f373"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:06.812275 13136 raft_consensus.cc:482] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:06.812798 13136 raft_consensus.cc:3037] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.812824 13007 raft_consensus.cc:2454] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.813297 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50561f1e0c5b47a78abb1fba84e725fa" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:06.813349 13071 leader_election.cc:302] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:06.814036 15123 raft_consensus.cc:2781] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:06.814251 15123 raft_consensus.cc:686] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:06.814788 15123 consensus_queue.cc:237] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:06.814690 13006 raft_consensus.cc:3037] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.818267 12877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dfa37b2ad2421b9242de5f0c227491" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:06.818523 12877 raft_consensus.cc:3032] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:06.818652 12877 raft_consensus.cc:729] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:06.818946 12877 consensus_queue.cc:260] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:06.819345 12877 raft_consensus.cc:3037] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.822083 13006 raft_consensus.cc:2454] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.822188 12877 raft_consensus.cc:2454] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.836032 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c3dd1267b741d4a5001ac7ae1d57ca" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:06.836269 12876 raft_consensus.cc:3037] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.840849 12877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00b9b3645a04a4686e5759bd8201eca" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:06.841117 12877 raft_consensus.cc:3037] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.843322 12745 raft_consensus.cc:2454] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.843905 12877 raft_consensus.cc:2454] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.845436 12876 raft_consensus.cc:2454] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.846916 13136 raft_consensus.cc:504] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:06.848047 13136 leader_election.cc:288] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:06.850068 13136 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9a992a1cf4484b298200677ee8b56424"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:06.850282 13136 raft_consensus.cc:482] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:06.850384 13136 raft_consensus.cc:3037] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.851099 15143 raft_consensus.cc:3032] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:06.851187 15143 raft_consensus.cc:729] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:06.851516 15143 consensus_queue.cc:260] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:06.852594 13134 raft_consensus.cc:1226] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Rejecting Update request from peer 4e9d799e45f8407590cb3096f5625c62 for earlier term 1. Current term is 2. Ops: []
I20240501 20:11:06.851917 15143 raft_consensus.cc:3037] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.854111 14586 consensus_queue.cc:1046] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:06.854380 14586 raft_consensus.cc:3032] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:06.854454 14586 raft_consensus.cc:729] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:06.855301 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04bef0a392e42ebae58a89d46e2f373" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:06.855574 12876 raft_consensus.cc:3032] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:06.855660 12876 raft_consensus.cc:729] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:06.855960 12876 consensus_queue.cc:260] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:06.856379 12876 raft_consensus.cc:3037] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.857909 13136 raft_consensus.cc:504] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:06.858505 13136 leader_election.cc:288] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:06.859400 12876 raft_consensus.cc:2454] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.859911 13072 leader_election.cc:302] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:06.860289 13072 leader_election.cc:302] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:06.860296 15139 raft_consensus.cc:2781] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:06.860570 15139 raft_consensus.cc:686] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:06.860651 15123 raft_consensus.cc:2781] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:06.860793 15123 raft_consensus.cc:686] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:06.854715 14586 consensus_queue.cc:260] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:06.860870 15139 consensus_queue.cc:237] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:06.861007 15123 consensus_queue.cc:237] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:06.861243 14586 raft_consensus.cc:3037] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.866463 13072 leader_election.cc:302] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:06.867662 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a992a1cf4484b298200677ee8b56424" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:06.868006 12876 raft_consensus.cc:3032] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:06.868165 12876 raft_consensus.cc:729] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:06.868553 12876 consensus_queue.cc:260] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:06.868595 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00b9b3645a04a4686e5759bd8201eca" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:06.869050 12876 raft_consensus.cc:3037] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.869427 12744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c3dd1267b741d4a5001ac7ae1d57ca" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:06.869628 12744 raft_consensus.cc:3032] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:06.869736 12744 raft_consensus.cc:729] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:06.870019 12744 consensus_queue.cc:260] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:06.870401 12744 raft_consensus.cc:3037] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.872540 12876 raft_consensus.cc:2454] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.873503 13072 leader_election.cc:302] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:06.873842 14527 raft_consensus.cc:2781] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:06.874025 12744 raft_consensus.cc:2454] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.874754 13134 raft_consensus.cc:1226] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Rejecting Update request from peer 77bf705599644326b6e71b5caf32fc7c for earlier term 1. Current term is 2. Ops: []
I20240501 20:11:06.875588 15140 raft_consensus.cc:2781] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:06.876487 12744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50561f1e0c5b47a78abb1fba84e725fa" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:06.877496 12743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70cd215b4f9445499e14c9fd18f7f7fe" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:06.877772 12743 raft_consensus.cc:3032] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:06.877923 12743 raft_consensus.cc:729] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:06.878293 12743 consensus_queue.cc:260] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:06.878413 12742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04bef0a392e42ebae58a89d46e2f373" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:06.878724 12742 raft_consensus.cc:3037] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.878829 12743 raft_consensus.cc:3037] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.882762 12742 raft_consensus.cc:2454] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.882764 12743 raft_consensus.cc:2454] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.884372 12743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a992a1cf4484b298200677ee8b56424" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:06.884606 12743 raft_consensus.cc:3037] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.884801 13071 leader_election.cc:302] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:06.888381 12743 raft_consensus.cc:2454] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.893011 15152 raft_consensus.cc:2781] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:06.894841 15152 raft_consensus.cc:686] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:06.895973 15152 consensus_queue.cc:237] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:06.893363 15140 raft_consensus.cc:686] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:06.898375 13134 raft_consensus.cc:1226] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Rejecting Update request from peer 4e9d799e45f8407590cb3096f5625c62 for earlier term 1. Current term is 2. Ops: []
I20240501 20:11:06.898284 15140 consensus_queue.cc:237] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:06.902835 12746 raft_consensus.cc:2454] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.909206 12744 raft_consensus.cc:2454] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.911459 13134 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b96f18923f9d41878cb01202e77be033"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:06.911669 13134 raft_consensus.cc:465] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 20:11:06.912624 13134 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5af8328d0e9248acb93e3707ffde7f8a"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:06.912837 13134 raft_consensus.cc:465] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 20:11:06.913709 13134 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e1dbb57f13d348cc996b5248b7623b8e"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:06.913924 13134 raft_consensus.cc:482] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:06.914090 13134 raft_consensus.cc:3037] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Advancing to term 2
W20240501 20:11:06.917304 12680 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.219.67:42701, user_credentials={real_user=slave}} blocked reactor thread for 51702us
I20240501 20:11:06.918079 14527 raft_consensus.cc:686] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:06.918535 14527 consensus_queue.cc:237] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:06.917963 13134 raft_consensus.cc:504] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:06.920192 13134 leader_election.cc:288] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:06.921094 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1dbb57f13d348cc996b5248b7623b8e" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:06.921270 12744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1dbb57f13d348cc996b5248b7623b8e" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:06.921389 13006 raft_consensus.cc:3037] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.921480 12744 raft_consensus.cc:3032] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:06.921587 12744 raft_consensus.cc:729] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:06.921888 12744 consensus_queue.cc:260] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:06.922268 12744 raft_consensus.cc:3037] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.924543 13006 raft_consensus.cc:2454] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.925014 12744 raft_consensus.cc:2454] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.925114 13071 leader_election.cc:302] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:06.925556 15151 raft_consensus.cc:2781] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:06.925773 15151 raft_consensus.cc:686] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:06.925733 13134 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3eac603886e4a84877d71ccebd0dff0"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:06.926035 13134 raft_consensus.cc:482] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:06.926142 13134 raft_consensus.cc:3037] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.926362 15151 consensus_queue.cc:237] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:06.929297 13134 raft_consensus.cc:504] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:06.930606 12744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3eac603886e4a84877d71ccebd0dff0" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:06.930855 12744 raft_consensus.cc:3032] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:06.930963 12744 raft_consensus.cc:729] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:06.931210 12744 consensus_queue.cc:260] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:06.931581 12744 raft_consensus.cc:3037] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.934284 12744 raft_consensus.cc:2454] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.934832 13072 leader_election.cc:302] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:06.935364 15125 raft_consensus.cc:2781] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:06.935603 15125 raft_consensus.cc:686] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:06.936023 15125 consensus_queue.cc:237] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:06.936923 12599 catalog_manager.cc:5484] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 1 to 2, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 2 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:06.937670 12599 catalog_manager.cc:5484] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 1 to 2, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 2 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:06.938274 12599 catalog_manager.cc:5484] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 1 to 2, leader changed from b4720929b44c4560bfdf08107e137362 (127.29.219.67) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 2 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:06.938894 12599 catalog_manager.cc:5484] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 1 to 2, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 2 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:06.939441 12599 catalog_manager.cc:5484] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 1 to 2, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 2 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:06.940034 12599 catalog_manager.cc:5484] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 1 to 2, leader changed from b4720929b44c4560bfdf08107e137362 (127.29.219.67) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 2 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:06.942353 13134 leader_election.cc:288] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 2 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:06.942772 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3eac603886e4a84877d71ccebd0dff0" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:06.942955 12876 raft_consensus.cc:3037] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:06.944785 12876 raft_consensus.cc:2454] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 2.
I20240501 20:11:06.953373 12599 catalog_manager.cc:5484] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 1 to 2, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 2 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:06.954033 12599 catalog_manager.cc:5484] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 1 to 2, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 2 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:06.954588 12599 catalog_manager.cc:5484] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 1 to 2, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 2 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:06.955121 12599 catalog_manager.cc:5484] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 1 to 2, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 2 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:06.955583 12599 catalog_manager.cc:5484] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 1 to 2, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 2 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:06.957247 12599 catalog_manager.cc:5484] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 1 to 2, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 2 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:06.957785 12599 catalog_manager.cc:5484] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 1 to 2, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 2 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:06.997701 12876 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e077004855949878d554a2053131790"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:06.997982 12876 raft_consensus.cc:482] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:06.998138 12876 raft_consensus.cc:3037] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.000511 12876 raft_consensus.cc:504] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.001375 12876 leader_election.cc:288] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:07.001945 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e077004855949878d554a2053131790" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.001966 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e077004855949878d554a2053131790" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.002162 13134 raft_consensus.cc:3032] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.002184 13006 raft_consensus.cc:3037] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.002275 13134 raft_consensus.cc:729] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:07.002594 13134 consensus_queue.cc:260] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.003157 13134 raft_consensus.cc:3037] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.004983 13006 raft_consensus.cc:2454] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 3.
I20240501 20:11:07.005897 12811 leader_election.cc:302] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:07.005975 13134 raft_consensus.cc:2396] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.006243 15013 raft_consensus.cc:2781] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.006408 15013 raft_consensus.cc:686] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.006739 15013 consensus_queue.cc:237] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.009472 12876 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eeba0c03d83a433d9a519ba9fe29b53a"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:07.009697 12876 raft_consensus.cc:482] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.009840 12876 raft_consensus.cc:3037] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.010548 12599 catalog_manager.cc:5484] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 2 to 3, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 3 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.012740 12876 raft_consensus.cc:504] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.015218 12876 leader_election.cc:288] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:07.015617 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeba0c03d83a433d9a519ba9fe29b53a" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.015846 13134 raft_consensus.cc:3032] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.015889 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeba0c03d83a433d9a519ba9fe29b53a" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.015974 13134 raft_consensus.cc:729] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:07.016108 13006 raft_consensus.cc:3037] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.016268 13134 consensus_queue.cc:260] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.016556 13134 raft_consensus.cc:3037] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.018895 13006 raft_consensus.cc:2454] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 3.
I20240501 20:11:07.019379 13134 raft_consensus.cc:2396] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.019393 12811 leader_election.cc:302] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:07.019713 15013 raft_consensus.cc:2781] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.019891 15013 raft_consensus.cc:686] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.020192 15013 consensus_queue.cc:237] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.020812 12876 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e6a39cb39f7c4559b97ffa749a658e7b"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:07.021046 12876 raft_consensus.cc:482] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.021185 12876 raft_consensus.cc:3037] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.023697 12599 catalog_manager.cc:5484] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 2 to 3, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 3 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.024621 12876 raft_consensus.cc:504] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.025379 12876 leader_election.cc:288] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:07.025642 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a39cb39f7c4559b97ffa749a658e7b" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.025795 13134 raft_consensus.cc:3032] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.025868 13134 raft_consensus.cc:729] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:07.026099 13134 consensus_queue.cc:260] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.026347 13134 raft_consensus.cc:3037] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.027247 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a39cb39f7c4559b97ffa749a658e7b" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.028043 13006 raft_consensus.cc:3037] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.029043 13134 raft_consensus.cc:2396] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.030122 12876 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3eac603886e4a84877d71ccebd0dff0"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:07.030367 12876 raft_consensus.cc:482] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.030755 12876 raft_consensus.cc:3037] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.031299 13006 raft_consensus.cc:2454] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 3.
I20240501 20:11:07.031736 12811 leader_election.cc:302] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters: bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:07.032085 15013 raft_consensus.cc:2781] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.032233 15013 raft_consensus.cc:686] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.032498 15013 consensus_queue.cc:237] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.034564 12876 raft_consensus.cc:504] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.035219 12876 leader_election.cc:288] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:07.035959 12599 catalog_manager.cc:5484] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 2 to 3, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 3 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.036235 12876 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ddb529dd5534f8bb66562e8467c1f12"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:07.036386 12876 raft_consensus.cc:465] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 20:11:07.037375 12876 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c3933feed404a4baee16428d2aafacf"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:07.037539 12876 raft_consensus.cc:482] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.037663 12876 raft_consensus.cc:3037] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.038892 12744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3eac603886e4a84877d71ccebd0dff0" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.039153 12744 raft_consensus.cc:3037] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.040505 12876 raft_consensus.cc:504] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.041525 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c3933feed404a4baee16428d2aafacf" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.041770 13136 raft_consensus.cc:3032] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.041850 13136 raft_consensus.cc:729] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:07.042078 12744 raft_consensus.cc:2454] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 3.
I20240501 20:11:07.042068 13136 consensus_queue.cc:260] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.042297 13136 raft_consensus.cc:3037] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.042645 12812 leader_election.cc:302] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters:
I20240501 20:11:07.042801 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3eac603886e4a84877d71ccebd0dff0" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.043025 13134 raft_consensus.cc:3032] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.043159 13134 raft_consensus.cc:729] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:07.043445 13134 consensus_queue.cc:260] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.043772 13134 raft_consensus.cc:3037] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.044721 13136 raft_consensus.cc:2396] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.045209 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c3933feed404a4baee16428d2aafacf" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.045431 13006 raft_consensus.cc:3037] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.046658 13134 raft_consensus.cc:2396] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.047791 13006 raft_consensus.cc:2454] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 3.
I20240501 20:11:07.048079 15013 raft_consensus.cc:2781] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.048280 15013 raft_consensus.cc:686] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.048249 12811 leader_election.cc:302] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters: bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:07.048519 14437 raft_consensus.cc:2781] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.048547 15013 consensus_queue.cc:237] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.049758 12876 leader_election.cc:288] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:07.050653 14437 raft_consensus.cc:686] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.050947 14437 consensus_queue.cc:237] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.051299 12876 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a8b101e9e38b4048bc86d76f97201534"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:07.051513 12876 raft_consensus.cc:482] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.051620 12876 raft_consensus.cc:3037] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.052394 12599 catalog_manager.cc:5484] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 2 to 3, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 3 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:07.054967 12876 raft_consensus.cc:504] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.055778 12876 leader_election.cc:288] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:07.056861 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b101e9e38b4048bc86d76f97201534" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.057116 13006 raft_consensus.cc:3037] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.058447 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b101e9e38b4048bc86d76f97201534" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.058727 13134 raft_consensus.cc:3032] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.058838 13134 raft_consensus.cc:729] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:07.059121 13134 consensus_queue.cc:260] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.059408 13134 raft_consensus.cc:3037] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.059834 13006 raft_consensus.cc:2454] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 3.
I20240501 20:11:07.060650 12811 leader_election.cc:302] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:07.060966 15013 raft_consensus.cc:2781] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.061092 15013 raft_consensus.cc:686] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.061364 15013 consensus_queue.cc:237] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.062927 13134 raft_consensus.cc:2396] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.063377 12876 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5f728b714ca142648ba5aa3459f7ae80"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:07.063611 12876 raft_consensus.cc:482] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.063822 12876 raft_consensus.cc:3037] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.066114 12599 catalog_manager.cc:5484] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 2 to 3, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 3 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.066563 12876 raft_consensus.cc:504] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.067312 12876 leader_election.cc:288] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:07.068560 12876 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "97dfa37b2ad2421b9242de5f0c227491"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:07.068827 12876 raft_consensus.cc:482] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.068991 12876 raft_consensus.cc:3037] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.069438 12744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f728b714ca142648ba5aa3459f7ae80" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.069701 12744 raft_consensus.cc:3037] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.071606 12744 raft_consensus.cc:2454] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 3.
I20240501 20:11:07.072019 12812 leader_election.cc:302] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters:
I20240501 20:11:07.072110 12876 raft_consensus.cc:504] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.072410 15165 raft_consensus.cc:2781] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.072700 15165 raft_consensus.cc:686] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.073081 15165 consensus_queue.cc:237] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.073272 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f728b714ca142648ba5aa3459f7ae80" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.073474 13134 raft_consensus.cc:3032] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.073451 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dfa37b2ad2421b9242de5f0c227491" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.073679 13136 raft_consensus.cc:3032] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.073777 13136 raft_consensus.cc:729] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:07.074018 13136 consensus_queue.cc:260] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.074411 13136 raft_consensus.cc:3037] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.075156 12599 catalog_manager.cc:5484] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 2 to 3, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 3 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.076886 12876 leader_election.cc:288] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:07.077013 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dfa37b2ad2421b9242de5f0c227491" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.077221 13006 raft_consensus.cc:3037] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.079413 13136 raft_consensus.cc:2396] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.079753 13006 raft_consensus.cc:2454] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 3.
I20240501 20:11:07.080047 13134 raft_consensus.cc:729] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:07.080384 13134 consensus_queue.cc:260] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.080741 13134 raft_consensus.cc:3037] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.080853 12811 leader_election.cc:302] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters: bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:07.081310 14437 raft_consensus.cc:2781] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.081459 14437 raft_consensus.cc:686] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.081790 14437 consensus_queue.cc:237] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.083209 12876 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c1211cfd40674438a14a49818863b3f0"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:07.083469 12876 raft_consensus.cc:482] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.083611 12876 raft_consensus.cc:3037] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.083639 13134 raft_consensus.cc:2396] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.087846 12876 raft_consensus.cc:504] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.088783 12876 leader_election.cc:288] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 2 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:07.089812 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1211cfd40674438a14a49818863b3f0" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.089505 12744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1211cfd40674438a14a49818863b3f0" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.090042 13006 raft_consensus.cc:3037] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.090166 12744 raft_consensus.cc:3032] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:07.090493 12876 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e00b9b3645a04a4686e5759bd8201eca"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:07.090269 12744 raft_consensus.cc:729] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.090725 12876 raft_consensus.cc:482] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.090828 12876 raft_consensus.cc:3037] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.091153 12744 consensus_queue.cc:260] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.091614 12744 raft_consensus.cc:3037] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.092208 12599 catalog_manager.cc:5484] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 2 to 3, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 3 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.092831 12599 catalog_manager.cc:5484] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 2 to 3, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 3 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.093740 12876 raft_consensus.cc:504] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.094422 12876 leader_election.cc:288] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:07.096062 12744 raft_consensus.cc:2454] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 2.
I20240501 20:11:07.096997 12812 leader_election.cc:302] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters:
I20240501 20:11:07.097334 14437 raft_consensus.cc:2781] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:07.097503 14437 raft_consensus.cc:686] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 2 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.097745 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00b9b3645a04a4686e5759bd8201eca" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.097815 14437 consensus_queue.cc:237] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.098006 13134 raft_consensus.cc:3032] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.098112 13134 raft_consensus.cc:729] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:07.098415 13134 consensus_queue.cc:260] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.098784 13134 raft_consensus.cc:3037] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.102483 13134 raft_consensus.cc:2396] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.104168 12744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00b9b3645a04a4686e5759bd8201eca" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.104408 12744 raft_consensus.cc:3037] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.096194 13006 raft_consensus.cc:2454] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 2.
I20240501 20:11:07.107600 12744 raft_consensus.cc:2454] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 3.
I20240501 20:11:07.107578 12877 raft_consensus.cc:1261] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Refusing update from remote peer bb1be87a7aee48a4b405b9d209c61231: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:11:07.108589 15151 consensus_queue.cc:1035] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.108716 12812 leader_election.cc:302] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters: bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:07.109694 15165 raft_consensus.cc:2781] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.109856 15165 raft_consensus.cc:686] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.110141 15165 consensus_queue.cc:237] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.111135 12599 catalog_manager.cc:5484] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 1 to 2, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 2 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:07.114794 12876 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "66c3dd1267b741d4a5001ac7ae1d57ca"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:07.115000 12876 raft_consensus.cc:482] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.115166 12876 raft_consensus.cc:3037] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.121510 12876 raft_consensus.cc:504] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.122826 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c3dd1267b741d4a5001ac7ae1d57ca" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.123054 13134 raft_consensus.cc:3032] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.123162 13134 raft_consensus.cc:729] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:07.123420 13134 consensus_queue.cc:260] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.123739 13134 raft_consensus.cc:3037] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.125972 14983 heartbeater.cc:503] Master 127.29.219.126:34159 was elected leader, sending a full tablet report...
I20240501 20:11:07.127220 13134 raft_consensus.cc:2396] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.128687 12744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c3dd1267b741d4a5001ac7ae1d57ca" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.128938 12744 raft_consensus.cc:3037] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.131731 12744 raft_consensus.cc:2454] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 3.
I20240501 20:11:07.132251 12812 leader_election.cc:302] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters: bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:07.133636 15165 raft_consensus.cc:2781] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.133733 12599 catalog_manager.cc:5484] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 2 to 3, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 3 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:07.134392 15165 raft_consensus.cc:686] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.134976 12876 leader_election.cc:288] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:07.134896 15165 consensus_queue.cc:237] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.139780 12876 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1d207517fea64fe1a83a2409157ae2bd"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:07.140332 12876 raft_consensus.cc:482] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.140717 12876 raft_consensus.cc:3037] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.141443 12599 catalog_manager.cc:5484] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 2 to 3, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 3 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.148093 12876 raft_consensus.cc:504] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.149837 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d207517fea64fe1a83a2409157ae2bd" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.149878 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d207517fea64fe1a83a2409157ae2bd" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.150120 13006 raft_consensus.cc:3037] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.150122 13134 raft_consensus.cc:3032] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:07.150239 13134 raft_consensus.cc:729] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:07.150489 12876 leader_election.cc:288] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 2 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:07.150575 13134 consensus_queue.cc:260] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.151091 13134 raft_consensus.cc:3037] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.151800 12874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b6ba30e296834fb28bf4a88141c24826"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:07.151988 12874 raft_consensus.cc:465] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 20:11:07.152974 12874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "17276a219da74d95919c8f427c9920c5"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:07.153010 13006 raft_consensus.cc:2454] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 2.
I20240501 20:11:07.153152 12874 raft_consensus.cc:482] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.153277 12874 raft_consensus.cc:3037] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.155268 13134 raft_consensus.cc:2454] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 2.
I20240501 20:11:07.155851 12812 leader_election.cc:302] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:07.155792 12874 raft_consensus.cc:504] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.156258 14437 raft_consensus.cc:2781] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:07.156432 14437 raft_consensus.cc:686] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [term 2 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.156808 14437 consensus_queue.cc:237] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.161933 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17276a219da74d95919c8f427c9920c5" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.162211 13006 raft_consensus.cc:3032] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:07.162302 13006 raft_consensus.cc:729] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:07.162586 13006 consensus_queue.cc:260] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.163000 13006 raft_consensus.cc:3037] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.163060 12744 raft_consensus.cc:1261] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Refusing update from remote peer bb1be87a7aee48a4b405b9d209c61231: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:11:07.163879 15125 consensus_queue.cc:1035] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.165913 13006 raft_consensus.cc:2454] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 2.
I20240501 20:11:07.166150 12874 leader_election.cc:288] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 2 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:07.166800 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17276a219da74d95919c8f427c9920c5" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.167061 12746 raft_consensus.cc:3037] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.172071 12811 leader_election.cc:302] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:07.172513 15165 raft_consensus.cc:2781] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:07.172714 15165 raft_consensus.cc:686] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 2 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.173096 15165 consensus_queue.cc:237] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.175406 12874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "248dfbfcb6214244a410083e8de22743"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:07.175626 12874 raft_consensus.cc:482] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.175679 12745 raft_consensus.cc:1261] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Refusing update from remote peer bb1be87a7aee48a4b405b9d209c61231: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:11:07.175732 12874 raft_consensus.cc:3037] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.176316 15125 consensus_queue.cc:1035] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.177632 12599 catalog_manager.cc:5484] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 1 to 2, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 2 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.178514 12874 raft_consensus.cc:504] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.179133 12874 leader_election.cc:288] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:07.179847 12743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "248dfbfcb6214244a410083e8de22743" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.180677 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "248dfbfcb6214244a410083e8de22743" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.180883 13134 raft_consensus.cc:3032] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.180986 13134 raft_consensus.cc:729] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:07.181257 13134 consensus_queue.cc:260] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.181574 13134 raft_consensus.cc:3037] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.181743 12874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "47b8c200bb05478db26c0d10b25a514c"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:07.181936 12874 raft_consensus.cc:482] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.182070 12874 raft_consensus.cc:3037] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.184381 13134 raft_consensus.cc:2454] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 3.
I20240501 20:11:07.184820 12812 leader_election.cc:302] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231; no voters: 4e9d799e45f8407590cb3096f5625c62
I20240501 20:11:07.175101 12746 raft_consensus.cc:2454] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 2.
I20240501 20:11:07.185215 15165 raft_consensus.cc:2781] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.185369 15165 raft_consensus.cc:686] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.185632 15165 consensus_queue.cc:237] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.186339 12874 raft_consensus.cc:504] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.187444 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b8c200bb05478db26c0d10b25a514c" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.187675 13006 raft_consensus.cc:3032] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:07.187860 13006 raft_consensus.cc:729] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:07.188239 13006 consensus_queue.cc:260] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.188591 13006 raft_consensus.cc:3037] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.189342 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b8c200bb05478db26c0d10b25a514c" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.189622 12746 raft_consensus.cc:3037] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.190253 12743 raft_consensus.cc:1261] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Refusing update from remote peer bb1be87a7aee48a4b405b9d209c61231: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:11:07.191417 15125 consensus_queue.cc:1035] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.193558 12746 raft_consensus.cc:2454] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 2.
I20240501 20:11:07.194077 12812 leader_election.cc:302] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters:
I20240501 20:11:07.194418 14437 raft_consensus.cc:2781] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:07.194572 14437 raft_consensus.cc:686] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 2 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.194975 13007 raft_consensus.cc:1261] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Refusing update from remote peer bb1be87a7aee48a4b405b9d209c61231: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:11:07.194856 14437 consensus_queue.cc:237] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.196269 15151 consensus_queue.cc:1035] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.198465 13005 raft_consensus.cc:1261] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Refusing update from remote peer bb1be87a7aee48a4b405b9d209c61231: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:11:07.201640 15125 consensus_queue.cc:1035] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.204905 12599 catalog_manager.cc:5484] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 2 to 3, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 3 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.205530 12599 catalog_manager.cc:5484] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 1 to 2, leader changed from b4720929b44c4560bfdf08107e137362 (127.29.219.67) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 2 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:07.208163 12874 leader_election.cc:288] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 2 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:07.211176 13006 raft_consensus.cc:2454] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 2.
I20240501 20:11:07.213907 12599 catalog_manager.cc:5484] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 1 to 2, leader changed from b4720929b44c4560bfdf08107e137362 (127.29.219.67) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 2 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:07.217671 12874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a04bef0a392e42ebae58a89d46e2f373"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:07.217926 12874 raft_consensus.cc:482] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.218075 12874 raft_consensus.cc:3037] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.223954 12746 raft_consensus.cc:1261] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Refusing update from remote peer bb1be87a7aee48a4b405b9d209c61231: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:11:07.225111 14527 consensus_queue.cc:1035] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.231540 13006 raft_consensus.cc:1261] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Refusing update from remote peer bb1be87a7aee48a4b405b9d209c61231: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:11:07.232318 14527 consensus_queue.cc:1035] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.233266 12874 raft_consensus.cc:504] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.234632 12874 leader_election.cc:288] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:07.235402 12743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04bef0a392e42ebae58a89d46e2f373" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.235406 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04bef0a392e42ebae58a89d46e2f373" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.235621 13134 raft_consensus.cc:3032] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.235652 12743 raft_consensus.cc:3037] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.235744 13134 raft_consensus.cc:729] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:07.236011 13134 consensus_queue.cc:260] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.236368 13134 raft_consensus.cc:3037] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.238490 12743 raft_consensus.cc:2454] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 3.
I20240501 20:11:07.239166 12812 leader_election.cc:302] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters:
I20240501 20:11:07.239517 14437 raft_consensus.cc:2781] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.239675 14437 raft_consensus.cc:686] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.239902 13134 raft_consensus.cc:2396] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.239997 14437 consensus_queue.cc:237] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.240644 12874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9a992a1cf4484b298200677ee8b56424"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:07.240854 12874 raft_consensus.cc:482] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.241111 12874 raft_consensus.cc:3037] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.245848 12874 raft_consensus.cc:504] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.247169 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a992a1cf4484b298200677ee8b56424" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.247381 13134 raft_consensus.cc:3032] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.247493 13134 raft_consensus.cc:729] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:07.247762 13134 consensus_queue.cc:260] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.248163 13134 raft_consensus.cc:3037] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.252046 13134 raft_consensus.cc:2396] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.253407 12743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a992a1cf4484b298200677ee8b56424" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.254906 12599 catalog_manager.cc:5484] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 2 to 3, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 3 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:07.256464 12743 raft_consensus.cc:3037] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.262092 12743 raft_consensus.cc:2454] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 3.
I20240501 20:11:07.262439 12874 leader_election.cc:288] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:07.262653 12812 leader_election.cc:302] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters: bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:07.262975 14437 raft_consensus.cc:2781] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.263164 14437 raft_consensus.cc:686] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.263515 12874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b96f18923f9d41878cb01202e77be033"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:07.263514 14437 consensus_queue.cc:237] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.263763 12874 raft_consensus.cc:482] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.263880 12874 raft_consensus.cc:3037] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.269796 12874 raft_consensus.cc:504] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.270848 12874 leader_election.cc:288] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 2 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:07.271417 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96f18923f9d41878cb01202e77be033" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.271675 13134 raft_consensus.cc:3032] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:07.271811 13134 raft_consensus.cc:729] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:07.272156 13134 consensus_queue.cc:260] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.272675 13134 raft_consensus.cc:3037] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.276557 13134 raft_consensus.cc:2454] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 2.
I20240501 20:11:07.277235 12812 leader_election.cc:302] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:07.277436 13004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96f18923f9d41878cb01202e77be033" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.277712 13004 raft_consensus.cc:3037] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.277890 15165 raft_consensus.cc:2781] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:07.278105 15165 raft_consensus.cc:686] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 2 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.278486 15165 consensus_queue.cc:237] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.281900 13004 raft_consensus.cc:2454] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 2.
I20240501 20:11:07.282001 12874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4488bf7917574118915a9355219528ee"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:07.282205 12874 raft_consensus.cc:465] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 20:11:07.283473 12874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d85b0f46583649e9bee3f433ba6f928e"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:07.283742 12874 raft_consensus.cc:465] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 20:11:07.284813 12874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e89e2dfe0abc487593a61f9f1b58ac65"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:07.284984 12874 raft_consensus.cc:482] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.285113 12874 raft_consensus.cc:3037] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.285636 12601 catalog_manager.cc:5484] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 2 to 3, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 3 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:07.288095 12874 raft_consensus.cc:504] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.288770 12874 leader_election.cc:288] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 2 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:07.289422 13004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89e2dfe0abc487593a61f9f1b58ac65" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.289646 13004 raft_consensus.cc:3032] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:07.289723 13004 raft_consensus.cc:729] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:07.289685 12743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89e2dfe0abc487593a61f9f1b58ac65" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.289927 12743 raft_consensus.cc:3037] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.289997 13004 consensus_queue.cc:260] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.290321 13004 raft_consensus.cc:3037] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.292730 12743 raft_consensus.cc:2454] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 2.
I20240501 20:11:07.292486 12601 catalog_manager.cc:5484] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 1 to 2, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 2 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.293002 13004 raft_consensus.cc:2454] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 2.
I20240501 20:11:07.293157 12812 leader_election.cc:302] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters:
I20240501 20:11:07.293519 14437 raft_consensus.cc:2781] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:07.293653 14437 raft_consensus.cc:686] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 2 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.293979 14437 consensus_queue.cc:237] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.300578 12601 catalog_manager.cc:5484] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 1 to 2, leader changed from b4720929b44c4560bfdf08107e137362 (127.29.219.67) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 2 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.326961 13004 raft_consensus.cc:1261] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Refusing update from remote peer bb1be87a7aee48a4b405b9d209c61231: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:11:07.327713 14527 consensus_queue.cc:1035] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.332775 12743 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9a992a1cf4484b298200677ee8b56424"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:07.333094 12743 raft_consensus.cc:482] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.333241 12743 raft_consensus.cc:3037] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.336241 12743 raft_consensus.cc:504] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.337724 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a992a1cf4484b298200677ee8b56424" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.337981 13134 raft_consensus.cc:3037] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.340873 13134 raft_consensus.cc:2396] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.342195 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a992a1cf4484b298200677ee8b56424" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.342448 12874 raft_consensus.cc:3032] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:07.342535 12874 raft_consensus.cc:729] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.342851 12874 consensus_queue.cc:260] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.343181 12874 raft_consensus.cc:3037] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.346007 12746 raft_consensus.cc:1261] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Refusing update from remote peer bb1be87a7aee48a4b405b9d209c61231: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:11:07.346096 12874 raft_consensus.cc:2396] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.346695 12681 leader_election.cc:302] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62; no voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:07.346699 15123 consensus_queue.cc:1035] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.347020 14586 raft_consensus.cc:2726] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:11:07.347277 12743 leader_election.cc:288] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 4 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:07.350788 12746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3eac603886e4a84877d71ccebd0dff0"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:07.350981 12746 raft_consensus.cc:482] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.351109 12746 raft_consensus.cc:3037] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.354442 12746 raft_consensus.cc:504] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.355914 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3eac603886e4a84877d71ccebd0dff0" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.356194 13134 raft_consensus.cc:3037] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.359040 13134 raft_consensus.cc:2396] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.359560 12746 leader_election.cc:288] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 4 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:07.359850 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3eac603886e4a84877d71ccebd0dff0" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.360095 12874 raft_consensus.cc:3032] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:07.360169 12874 raft_consensus.cc:729] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.360467 12874 consensus_queue.cc:260] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.360761 12874 raft_consensus.cc:3037] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.362845 12874 raft_consensus.cc:2396] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.363325 12681 leader_election.cc:302] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62; no voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:07.363794 14586 raft_consensus.cc:2726] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:11:07.363989 12746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f22d4331ed6f4e86b85a9919c219ab84"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:07.364185 12746 raft_consensus.cc:482] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.364281 12746 raft_consensus.cc:3037] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.366176 12746 raft_consensus.cc:504] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.366873 12746 leader_election.cc:288] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 3 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:07.367484 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22d4331ed6f4e86b85a9919c219ab84" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.367714 13134 raft_consensus.cc:3032] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.367735 13004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22d4331ed6f4e86b85a9919c219ab84" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.367828 13134 raft_consensus.cc:729] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:07.367950 13004 raft_consensus.cc:3037] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.368552 13134 consensus_queue.cc:260] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.368983 13134 raft_consensus.cc:3037] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.369575 12746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ddb529dd5534f8bb66562e8467c1f12"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:07.369833 12746 raft_consensus.cc:482] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.370016 12746 raft_consensus.cc:3037] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.370823 13004 raft_consensus.cc:2454] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 3.
I20240501 20:11:07.371325 12680 leader_election.cc:302] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:07.371708 14586 raft_consensus.cc:2781] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.371753 13134 raft_consensus.cc:2454] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 3.
I20240501 20:11:07.371886 14586 raft_consensus.cc:686] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.372164 14586 consensus_queue.cc:237] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.373075 12746 raft_consensus.cc:504] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.374651 12746 leader_election.cc:288] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 2 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:07.375032 13004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ddb529dd5534f8bb66562e8467c1f12" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.375267 13004 raft_consensus.cc:3037] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.375725 12743 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "436330ea0516443799906257122b38af"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:07.375916 12743 raft_consensus.cc:482] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.376008 12743 raft_consensus.cc:3037] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.375690 12599 catalog_manager.cc:5484] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 2 to 3, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 3 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.377857 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ddb529dd5534f8bb66562e8467c1f12" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.378046 13004 raft_consensus.cc:2454] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 2.
I20240501 20:11:07.378163 12874 raft_consensus.cc:3032] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:07.378262 12874 raft_consensus.cc:729] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.378469 12680 leader_election.cc:302] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:07.378664 12743 raft_consensus.cc:504] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.378830 15149 raft_consensus.cc:2781] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:07.378993 15149 raft_consensus.cc:686] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 2 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.379400 12743 leader_election.cc:288] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 3 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:07.379464 15149 consensus_queue.cc:237] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.379853 13004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436330ea0516443799906257122b38af" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.380028 13004 raft_consensus.cc:3037] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.380472 12744 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "66c3dd1267b741d4a5001ac7ae1d57ca"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:07.380661 12744 raft_consensus.cc:482] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.380771 12744 raft_consensus.cc:3037] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.381172 12874 consensus_queue.cc:260] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.381747 12874 raft_consensus.cc:3037] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.382907 13004 raft_consensus.cc:2454] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 3.
I20240501 20:11:07.383416 12680 leader_election.cc:302] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:07.383396 12744 raft_consensus.cc:504] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.383827 15150 raft_consensus.cc:2781] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.383967 15150 raft_consensus.cc:686] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.384040 12744 leader_election.cc:288] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 4 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:07.384292 15150 consensus_queue.cc:237] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.385049 12745 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "50561f1e0c5b47a78abb1fba84e725fa"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:07.385278 12745 raft_consensus.cc:482] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.385425 12745 raft_consensus.cc:3037] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.387642 12874 raft_consensus.cc:2454] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 2.
I20240501 20:11:07.390539 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436330ea0516443799906257122b38af" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.390842 13134 raft_consensus.cc:3032] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.390983 13134 raft_consensus.cc:729] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:07.391335 13134 consensus_queue.cc:260] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.391834 13134 raft_consensus.cc:3037] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.394933 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c3dd1267b741d4a5001ac7ae1d57ca" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.395174 13136 raft_consensus.cc:3037] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.395815 12745 raft_consensus.cc:504] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.398733 13136 raft_consensus.cc:2396] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.400326 12599 catalog_manager.cc:5484] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 1 to 2, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 2 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.400902 13134 raft_consensus.cc:2454] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 3.
I20240501 20:11:07.400897 12599 catalog_manager.cc:5484] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 2 to 3, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 3 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.403236 12745 leader_election.cc:288] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 3 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:07.403817 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50561f1e0c5b47a78abb1fba84e725fa" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.403802 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c3dd1267b741d4a5001ac7ae1d57ca" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.404002 13134 raft_consensus.cc:3032] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.404036 12874 raft_consensus.cc:3032] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:07.404109 13134 raft_consensus.cc:729] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:07.404138 12874 raft_consensus.cc:729] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.404256 13004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50561f1e0c5b47a78abb1fba84e725fa" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.404384 12746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5f728b714ca142648ba5aa3459f7ae80"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:07.404479 13004 raft_consensus.cc:3037] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.404352 13134 consensus_queue.cc:260] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.404408 12874 consensus_queue.cc:260] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.404589 12746 raft_consensus.cc:482] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.404685 12746 raft_consensus.cc:3037] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.404678 13134 raft_consensus.cc:3037] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.404683 12874 raft_consensus.cc:3037] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.407289 13004 raft_consensus.cc:2454] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 3.
I20240501 20:11:07.407311 12746 raft_consensus.cc:504] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.407680 12680 leader_election.cc:302] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:07.407874 13134 raft_consensus.cc:2396] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.407922 15149 raft_consensus.cc:2781] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.408080 15149 raft_consensus.cc:686] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.408233 12874 raft_consensus.cc:2396] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.408360 15149 consensus_queue.cc:237] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.409121 12681 leader_election.cc:302] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62; no voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:07.409386 14586 raft_consensus.cc:2726] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:11:07.409477 12746 leader_election.cc:288] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 4 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:07.409904 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f728b714ca142648ba5aa3459f7ae80" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.410079 13134 raft_consensus.cc:3037] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.410640 12746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c1211cfd40674438a14a49818863b3f0"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:07.411149 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f728b714ca142648ba5aa3459f7ae80" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.411347 12874 raft_consensus.cc:3032] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:07.411417 12874 raft_consensus.cc:729] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.411701 12874 consensus_queue.cc:260] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.411978 12874 raft_consensus.cc:3037] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.412000 12599 catalog_manager.cc:5484] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 2 to 3, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 3 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.410871 12746 raft_consensus.cc:482] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.413373 12746 raft_consensus.cc:3037] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.414522 13134 raft_consensus.cc:2396] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.415361 12874 raft_consensus.cc:2396] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.415742 12681 leader_election.cc:302] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62; no voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:07.416054 15149 raft_consensus.cc:2726] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:11:07.417058 12746 raft_consensus.cc:504] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.418154 13004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1211cfd40674438a14a49818863b3f0" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.418365 13004 raft_consensus.cc:3037] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.417693 12746 leader_election.cc:288] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 3 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:07.419155 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1211cfd40674438a14a49818863b3f0" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.419507 12874 raft_consensus.cc:3032] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.420080 12746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e00b9b3645a04a4686e5759bd8201eca"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:07.420253 12746 raft_consensus.cc:482] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.420338 12746 raft_consensus.cc:3037] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.421353 13004 raft_consensus.cc:2454] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 3.
I20240501 20:11:07.419596 12874 raft_consensus.cc:729] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.421696 12874 consensus_queue.cc:260] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.422108 12874 raft_consensus.cc:3037] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.425503 12680 leader_election.cc:302] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:07.425581 12874 raft_consensus.cc:2396] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.425851 15149 raft_consensus.cc:2781] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.426040 15149 raft_consensus.cc:686] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.426687 15149 consensus_queue.cc:237] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.427961 12746 raft_consensus.cc:504] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.428294 13004 raft_consensus.cc:1261] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Refusing update from remote peer 77bf705599644326b6e71b5caf32fc7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 20:11:07.428608 12746 leader_election.cc:288] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 4 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:07.428853 14437 consensus_queue.cc:1035] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.429322 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00b9b3645a04a4686e5759bd8201eca" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.429546 13134 raft_consensus.cc:3037] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.430433 12746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b6ba30e296834fb28bf4a88141c24826"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:07.430670 12746 raft_consensus.cc:482] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.430884 12746 raft_consensus.cc:3037] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.432423 13134 raft_consensus.cc:2396] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.433272 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00b9b3645a04a4686e5759bd8201eca" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.433501 12874 raft_consensus.cc:3032] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:07.433596 12874 raft_consensus.cc:729] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.433924 12874 consensus_queue.cc:260] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.434244 12874 raft_consensus.cc:3037] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.434655 12599 catalog_manager.cc:5484] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 2 to 3, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 3 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.435129 12746 raft_consensus.cc:504] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.436170 12877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ba30e296834fb28bf4a88141c24826" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.436383 12877 raft_consensus.cc:3032] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:07.436471 12877 raft_consensus.cc:729] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.436713 12877 consensus_queue.cc:260] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.437079 12874 raft_consensus.cc:2396] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.437137 12877 raft_consensus.cc:3037] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.439210 12877 raft_consensus.cc:2454] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 2.
I20240501 20:11:07.440704 12681 leader_election.cc:302] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62; no voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:07.441107 15149 raft_consensus.cc:2726] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:11:07.441311 12681 leader_election.cc:302] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters:
I20240501 20:11:07.441618 15149 raft_consensus.cc:2781] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:07.441825 15149 raft_consensus.cc:686] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 2 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.442202 12746 leader_election.cc:288] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 2 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:07.442163 15149 consensus_queue.cc:237] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.443403 12746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dafd65d56bb243c78cb4490d7da7b18a"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:07.443612 12746 raft_consensus.cc:482] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.443495 13004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ba30e296834fb28bf4a88141c24826" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.443746 12746 raft_consensus.cc:3037] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.443723 13007 raft_consensus.cc:1261] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Refusing update from remote peer 77bf705599644326b6e71b5caf32fc7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 20:11:07.443933 13004 raft_consensus.cc:3037] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.444356 14437 consensus_queue.cc:1035] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.451086 13004 raft_consensus.cc:2454] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 2.
I20240501 20:11:07.451999 12746 raft_consensus.cc:504] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.453547 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dafd65d56bb243c78cb4490d7da7b18a" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.453768 13134 raft_consensus.cc:3032] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.453876 13134 raft_consensus.cc:729] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:07.454154 13134 consensus_queue.cc:260] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.454579 13134 raft_consensus.cc:3037] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.454509 13004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dafd65d56bb243c78cb4490d7da7b18a" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.455076 13004 raft_consensus.cc:3037] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.457891 13134 raft_consensus.cc:2454] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 3.
I20240501 20:11:07.458527 12681 leader_election.cc:302] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:07.459051 12746 leader_election.cc:288] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 3 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:07.459061 14586 raft_consensus.cc:2781] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.459317 14586 raft_consensus.cc:686] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.459760 14586 consensus_queue.cc:237] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.462597 12746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "17276a219da74d95919c8f427c9920c5"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:07.462970 12746 raft_consensus.cc:482] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.463232 12746 raft_consensus.cc:3037] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.470547 13004 raft_consensus.cc:2454] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 3.
I20240501 20:11:07.471704 12601 catalog_manager.cc:5484] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 1 to 2, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 2 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.472779 12746 raft_consensus.cc:504] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.473174 13134 raft_consensus.cc:1261] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Refusing update from remote peer 77bf705599644326b6e71b5caf32fc7c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 20:11:07.473304 13134 pending_rounds.cc:76] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231: Aborting all ops after (but not including) 3
I20240501 20:11:07.473398 13134 pending_rounds.cc:97] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 20:11:07.473541 13134 raft_consensus.cc:2866] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:11:07.474188 15013 consensus_queue.cc:1035] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.474900 13004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17276a219da74d95919c8f427c9920c5" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.475364 13004 raft_consensus.cc:3037] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.478497 13004 raft_consensus.cc:2454] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 3.
I20240501 20:11:07.479002 12680 leader_election.cc:302] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:07.479521 15149 raft_consensus.cc:2781] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.479672 15149 raft_consensus.cc:686] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.479467 12877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17276a219da74d95919c8f427c9920c5" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.479835 12877 raft_consensus.cc:3032] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.479913 12877 raft_consensus.cc:729] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.479969 15149 consensus_queue.cc:237] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.480158 12877 consensus_queue.cc:260] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.480928 12877 raft_consensus.cc:3037] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.479005 12746 leader_election.cc:288] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 3 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:07.482738 12746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "248dfbfcb6214244a410083e8de22743"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:07.482947 12746 raft_consensus.cc:482] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.483038 12746 raft_consensus.cc:3037] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.483784 12877 raft_consensus.cc:2396] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.485375 12599 catalog_manager.cc:5484] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 2 to 3, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 3 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.487483 12746 raft_consensus.cc:504] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.488158 12746 leader_election.cc:288] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 3 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:07.489305 12745 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "47b8c200bb05478db26c0d10b25a514c"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:07.489280 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "248dfbfcb6214244a410083e8de22743" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.489502 12745 raft_consensus.cc:482] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.489599 12874 raft_consensus.cc:2379] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 in current term 3: Already voted for candidate 77bf705599644326b6e71b5caf32fc7c in this term.
I20240501 20:11:07.489607 12745 raft_consensus.cc:3037] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.489694 13136 raft_consensus.cc:1261] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Refusing update from remote peer 77bf705599644326b6e71b5caf32fc7c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 20:11:07.489826 13136 pending_rounds.cc:76] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231: Aborting all ops after (but not including) 3
I20240501 20:11:07.489884 13136 pending_rounds.cc:97] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 20:11:07.489964 13136 raft_consensus.cc:2866] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:11:07.490767 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "248dfbfcb6214244a410083e8de22743" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.491012 13134 raft_consensus.cc:2379] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 in current term 3: Already voted for candidate 77bf705599644326b6e71b5caf32fc7c in this term.
I20240501 20:11:07.491286 13007 raft_consensus.cc:1261] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Refusing update from remote peer bb1be87a7aee48a4b405b9d209c61231: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:11:07.491945 14437 consensus_queue.cc:1035] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.492175 12681 leader_election.cc:302] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62; no voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:07.492312 12745 raft_consensus.cc:504] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.492424 15152 consensus_queue.cc:1035] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.492615 15150 raft_consensus.cc:2726] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 20:11:07.492997 12745 leader_election.cc:288] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 3 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:07.494109 12743 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a04bef0a392e42ebae58a89d46e2f373"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:07.494295 12743 raft_consensus.cc:482] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.494429 12743 raft_consensus.cc:3037] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.494843 13007 raft_consensus.cc:1261] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Refusing update from remote peer 77bf705599644326b6e71b5caf32fc7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 20:11:07.495421 14437 consensus_queue.cc:1035] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.499233 12743 raft_consensus.cc:504] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.499962 12743 leader_election.cc:288] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 4 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:07.500126 12877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b8c200bb05478db26c0d10b25a514c" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.500407 12877 raft_consensus.cc:3032] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.500515 12877 raft_consensus.cc:729] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.500806 12877 consensus_queue.cc:260] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.501165 12877 raft_consensus.cc:3037] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.501178 12744 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4488bf7917574118915a9355219528ee"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:07.501443 12744 raft_consensus.cc:482] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.501580 12744 raft_consensus.cc:3037] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.501891 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04bef0a392e42ebae58a89d46e2f373" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.502079 12874 raft_consensus.cc:3032] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:07.502157 12874 raft_consensus.cc:729] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.502444 12874 consensus_queue.cc:260] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.502727 12874 raft_consensus.cc:3037] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.504456 12744 raft_consensus.cc:504] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.505563 13005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4488bf7917574118915a9355219528ee" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.505611 12874 raft_consensus.cc:2396] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.505774 13005 raft_consensus.cc:3037] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.507104 12877 raft_consensus.cc:2396] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.509477 13005 raft_consensus.cc:2454] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 2.
I20240501 20:11:07.510043 12680 leader_election.cc:302] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:07.510413 15150 raft_consensus.cc:2781] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:07.510543 15150 raft_consensus.cc:686] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 2 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.510677 12744 leader_election.cc:288] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 2 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:07.510859 15150 consensus_queue.cc:237] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.511898 12743 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "70cd215b4f9445499e14c9fd18f7f7fe"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:07.512141 12743 raft_consensus.cc:482] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.512279 12743 raft_consensus.cc:3037] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.513286 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04bef0a392e42ebae58a89d46e2f373" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.513517 13134 raft_consensus.cc:3037] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.513736 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b8c200bb05478db26c0d10b25a514c" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.514020 13006 raft_consensus.cc:3037] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.515408 12743 raft_consensus.cc:504] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.516773 13007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70cd215b4f9445499e14c9fd18f7f7fe" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.516963 13007 raft_consensus.cc:3037] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.518824 13136 raft_consensus.cc:1261] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Refusing update from remote peer 77bf705599644326b6e71b5caf32fc7c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 20:11:07.518934 13136 pending_rounds.cc:76] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231: Aborting all ops after (but not including) 3
I20240501 20:11:07.518983 13136 pending_rounds.cc:97] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 20:11:07.519052 13136 raft_consensus.cc:2866] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:11:07.520027 15165 consensus_queue.cc:1035] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.521361 13003 raft_consensus.cc:1261] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Refusing update from remote peer 77bf705599644326b6e71b5caf32fc7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 20:11:07.521423 13007 raft_consensus.cc:2454] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 3.
I20240501 20:11:07.521831 12680 leader_election.cc:302] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:07.522137 14586 raft_consensus.cc:2781] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.522112 15013 consensus_queue.cc:1035] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.522284 14586 raft_consensus.cc:686] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.522528 14586 consensus_queue.cc:237] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.523998 13003 raft_consensus.cc:1261] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Refusing update from remote peer 77bf705599644326b6e71b5caf32fc7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 20:11:07.524195 12743 leader_election.cc:288] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 3 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:07.524709 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70cd215b4f9445499e14c9fd18f7f7fe" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.524811 15165 consensus_queue.cc:1035] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.524945 13136 raft_consensus.cc:3032] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.525035 13136 raft_consensus.cc:729] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:07.525333 13136 consensus_queue.cc:260] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.525539 13134 raft_consensus.cc:2396] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.525715 13136 raft_consensus.cc:3037] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.526091 12681 leader_election.cc:302] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62; no voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:07.526599 15149 raft_consensus.cc:2726] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:11:07.527454 12601 catalog_manager.cc:5484] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 2 to 3, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 3 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.528733 13136 raft_consensus.cc:2454] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 3.
I20240501 20:11:07.529577 12743 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d85b0f46583649e9bee3f433ba6f928e"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:07.529807 12743 raft_consensus.cc:482] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.529959 12743 raft_consensus.cc:3037] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.532629 12877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4488bf7917574118915a9355219528ee" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.533337 12877 raft_consensus.cc:3032] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:07.533679 12877 raft_consensus.cc:729] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.533963 12743 raft_consensus.cc:504] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.534478 12877 consensus_queue.cc:260] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.535136 13003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85b0f46583649e9bee3f433ba6f928e" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.535372 12877 raft_consensus.cc:3037] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.535377 13003 raft_consensus.cc:3037] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.538200 13003 raft_consensus.cc:2454] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 2.
I20240501 20:11:07.538677 12680 leader_election.cc:302] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:07.538941 13136 raft_consensus.cc:1261] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Refusing update from remote peer 77bf705599644326b6e71b5caf32fc7c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 20:11:07.539059 13136 pending_rounds.cc:76] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231: Aborting all ops after (but not including) 3
I20240501 20:11:07.539065 15149 raft_consensus.cc:2781] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:07.539119 13136 pending_rounds.cc:97] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 20:11:07.539197 13136 raft_consensus.cc:2866] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:11:07.539290 15149 raft_consensus.cc:686] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [term 2 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.539575 15149 consensus_queue.cc:237] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.539835 14437 consensus_queue.cc:1035] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.539964 13006 raft_consensus.cc:2454] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 3.
I20240501 20:11:07.540439 12680 leader_election.cc:302] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters: 77bf705599644326b6e71b5caf32fc7c
I20240501 20:11:07.540881 14586 raft_consensus.cc:2781] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.541024 14586 raft_consensus.cc:686] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.541286 14586 consensus_queue.cc:237] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.543057 12743 leader_election.cc:288] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 2 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:07.543501 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85b0f46583649e9bee3f433ba6f928e" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.543743 12874 raft_consensus.cc:3032] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:07.543828 12874 raft_consensus.cc:729] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.544108 12874 consensus_queue.cc:260] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.544602 12874 raft_consensus.cc:3037] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.548939 12874 raft_consensus.cc:2454] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 2.
I20240501 20:11:07.549309 13136 raft_consensus.cc:1261] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Refusing update from remote peer 77bf705599644326b6e71b5caf32fc7c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 20:11:07.549492 13136 pending_rounds.cc:76] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231: Aborting all ops after (but not including) 3
I20240501 20:11:07.549573 13136 pending_rounds.cc:97] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 20:11:07.549674 13136 raft_consensus.cc:2866] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:11:07.551208 15165 consensus_queue.cc:1035] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.552788 12746 raft_consensus.cc:1261] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Refusing update from remote peer bb1be87a7aee48a4b405b9d209c61231: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:11:07.554029 15152 consensus_queue.cc:1035] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.555258 12743 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e89e2dfe0abc487593a61f9f1b58ac65"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:07.555542 12743 raft_consensus.cc:482] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.555678 12743 raft_consensus.cc:3037] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.561116 12743 raft_consensus.cc:504] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.561851 13136 raft_consensus.cc:1261] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Refusing update from remote peer 77bf705599644326b6e71b5caf32fc7c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 20:11:07.561950 12743 leader_election.cc:288] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 3 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:07.562011 13136 pending_rounds.cc:76] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231: Aborting all ops after (but not including) 3
I20240501 20:11:07.562077 13136 pending_rounds.cc:97] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 20:11:07.562186 13136 raft_consensus.cc:2866] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:11:07.562839 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89e2dfe0abc487593a61f9f1b58ac65" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.563088 12874 raft_consensus.cc:3032] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.563035 12744 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5af8328d0e9248acb93e3707ffde7f8a"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:07.563169 12874 raft_consensus.cc:729] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.563256 12744 raft_consensus.cc:482] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.563369 12744 raft_consensus.cc:3037] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.563501 12874 consensus_queue.cc:260] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.563861 12874 raft_consensus.cc:3037] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.566372 12744 raft_consensus.cc:504] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.567147 12744 leader_election.cc:288] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 2 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:07.568084 12877 raft_consensus.cc:2454] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 2.
I20240501 20:11:07.568166 12744 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e1dbb57f13d348cc996b5248b7623b8e"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:07.568352 12744 raft_consensus.cc:482] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.568488 12744 raft_consensus.cc:3037] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.568521 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af8328d0e9248acb93e3707ffde7f8a" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.568749 13136 raft_consensus.cc:3032] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 20:11:07.568841 13136 raft_consensus.cc:729] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:07.569133 13136 consensus_queue.cc:260] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.569530 13136 raft_consensus.cc:3037] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.570822 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af8328d0e9248acb93e3707ffde7f8a" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.571048 13006 raft_consensus.cc:3037] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362 [term 1 FOLLOWER]: Advancing to term 2
I20240501 20:11:07.571333 12744 raft_consensus.cc:504] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.572367 13003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1dbb57f13d348cc996b5248b7623b8e" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.573590 12746 raft_consensus.cc:1226] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Rejecting Update request from peer bb1be87a7aee48a4b405b9d209c61231 for earlier term 2. Current term is 3. Ops: []
I20240501 20:11:07.573556 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1dbb57f13d348cc996b5248b7623b8e" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.573773 13134 raft_consensus.cc:3032] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.573776 13006 raft_consensus.cc:2454] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 2.
I20240501 20:11:07.573875 13134 raft_consensus.cc:729] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:07.574144 13134 consensus_queue.cc:260] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.574317 12680 leader_election.cc:302] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:07.574399 13134 raft_consensus.cc:3037] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.574728 15150 raft_consensus.cc:2781] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 20:11:07.574932 15150 raft_consensus.cc:686] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [term 2 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.574877 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89e2dfe0abc487593a61f9f1b58ac65" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.575119 13002 raft_consensus.cc:3037] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.575268 15150 consensus_queue.cc:237] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.577258 13134 raft_consensus.cc:2454] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 3.
I20240501 20:11:07.577694 12681 leader_election.cc:302] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, bb1be87a7aee48a4b405b9d209c61231; no voters: b4720929b44c4560bfdf08107e137362
I20240501 20:11:07.577824 13002 raft_consensus.cc:2454] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 3.
I20240501 20:11:07.578202 12680 leader_election.cc:302] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:07.578512 15143 raft_consensus.cc:2781] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.578552 12744 leader_election.cc:288] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 3 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:07.579457 15149 raft_consensus.cc:2781] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.579820 15149 raft_consensus.cc:686] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.580291 15149 consensus_queue.cc:237] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.582922 15165 consensus_queue.cc:1035] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.587688 12874 raft_consensus.cc:2396] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.590294 13136 raft_consensus.cc:2454] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 2.
I20240501 20:11:07.594420 13002 raft_consensus.cc:1261] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Refusing update from remote peer 77bf705599644326b6e71b5caf32fc7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 20:11:07.595085 14437 consensus_queue.cc:1035] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.598299 12601 catalog_manager.cc:5484] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 1 to 2, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 2 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.599514 13134 raft_consensus.cc:1261] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Refusing update from remote peer 77bf705599644326b6e71b5caf32fc7c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 20:11:07.600389 14437 consensus_queue.cc:1035] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:07.604508 15114 heartbeater.cc:503] Master 127.29.219.126:34159 was elected leader, sending a full tablet report...
I20240501 20:11:07.607867 12743 raft_consensus.cc:1261] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Refusing update from remote peer 77bf705599644326b6e71b5caf32fc7c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 20:11:07.608850 15165 consensus_queue.cc:1035] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:07.610224 12601 catalog_manager.cc:5484] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 2 to 3, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 3 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.613509 15143 raft_consensus.cc:686] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.613998 15143 consensus_queue.cc:237] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.624272 12599 catalog_manager.cc:5484] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 1 to 2, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 2 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:07.624929 12599 catalog_manager.cc:5484] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 2 to 3, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 3 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.625473 12599 catalog_manager.cc:5484] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 2 to 3, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 3 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.625958 12599 catalog_manager.cc:5484] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 1 to 2, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 2 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.626538 12599 catalog_manager.cc:5484] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 2 to 3, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 3 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:07.633353 13004 raft_consensus.cc:1261] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Refusing update from remote peer 77bf705599644326b6e71b5caf32fc7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:11:07.633900 15165 consensus_queue.cc:1035] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.640120 13136 raft_consensus.cc:1261] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Refusing update from remote peer 77bf705599644326b6e71b5caf32fc7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 20:11:07.640858 14437 consensus_queue.cc:1035] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:07.651250 13004 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e077004855949878d554a2053131790"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:07.651536 13004 raft_consensus.cc:482] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.651669 13004 raft_consensus.cc:3037] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.654594 13004 raft_consensus.cc:504] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.656535 13004 leader_election.cc:288] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:07.656970 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e077004855949878d554a2053131790" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.656970 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e077004855949878d554a2053131790" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.657208 12874 raft_consensus.cc:3032] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:07.657214 13134 raft_consensus.cc:3037] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.657310 12874 raft_consensus.cc:729] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.657613 12874 consensus_queue.cc:260] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.657920 12874 raft_consensus.cc:3037] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.659966 13134 raft_consensus.cc:2454] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 4.
I20240501 20:11:07.660081 12874 raft_consensus.cc:2454] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 4.
I20240501 20:11:07.660734 12942 leader_election.cc:302] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:07.661624 14590 raft_consensus.cc:2781] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:11:07.663363 14590 raft_consensus.cc:686] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:07.663800 14590 consensus_queue.cc:237] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.667659 12599 catalog_manager.cc:5484] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 3 to 4, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 4 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.668196 13004 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c1211cfd40674438a14a49818863b3f0"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:07.668500 13004 raft_consensus.cc:482] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.668686 13004 raft_consensus.cc:3037] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.671957 13004 raft_consensus.cc:504] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.672899 13004 leader_election.cc:288] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:07.673365 12743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1211cfd40674438a14a49818863b3f0" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.673595 12743 raft_consensus.cc:3032] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:07.673684 12743 raft_consensus.cc:729] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.673717 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1211cfd40674438a14a49818863b3f0" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.673967 12874 raft_consensus.cc:3037] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.673980 12743 consensus_queue.cc:260] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.674294 12743 raft_consensus.cc:3037] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.676832 12743 raft_consensus.cc:2396] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.676856 12874 raft_consensus.cc:2396] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.677470 12942 leader_election.cc:302] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4720929b44c4560bfdf08107e137362; no voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c
I20240501 20:11:07.677894 15250 raft_consensus.cc:2726] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:11:07.678253 13004 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eeba0c03d83a433d9a519ba9fe29b53a"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:07.678507 13004 raft_consensus.cc:482] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.678714 13004 raft_consensus.cc:3037] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.682899 13004 raft_consensus.cc:504] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.683713 13004 leader_election.cc:288] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:07.684504 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeba0c03d83a433d9a519ba9fe29b53a" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.684746 13136 raft_consensus.cc:3037] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.684859 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeba0c03d83a433d9a519ba9fe29b53a" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.685091 12874 raft_consensus.cc:3032] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:07.685201 12874 raft_consensus.cc:729] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.685480 12874 consensus_queue.cc:260] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.685890 12874 raft_consensus.cc:3037] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.687474 13004 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e6a39cb39f7c4559b97ffa749a658e7b"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:07.687592 13136 raft_consensus.cc:2454] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 4.
I20240501 20:11:07.687662 13004 raft_consensus.cc:482] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.687772 13004 raft_consensus.cc:3037] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.688011 12942 leader_election.cc:302] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:07.688359 15250 raft_consensus.cc:2781] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:11:07.688539 15250 raft_consensus.cc:686] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:07.688757 12874 raft_consensus.cc:2454] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 4.
I20240501 20:11:07.688831 15250 consensus_queue.cc:237] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.691804 12604 catalog_manager.cc:5484] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 3 to 4, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 4 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:07.692023 13004 raft_consensus.cc:504] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.692579 13004 leader_election.cc:288] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:07.693306 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a39cb39f7c4559b97ffa749a658e7b" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.693558 13136 raft_consensus.cc:3037] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.696177 13136 raft_consensus.cc:2454] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 4.
I20240501 20:11:07.696496 13004 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a8b101e9e38b4048bc86d76f97201534"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:07.696683 13004 raft_consensus.cc:482] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.696816 13004 raft_consensus.cc:3037] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.697530 12942 leader_election.cc:302] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:07.697865 15250 raft_consensus.cc:2781] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:11:07.698119 15250 raft_consensus.cc:686] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:07.698421 15250 consensus_queue.cc:237] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.698964 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a39cb39f7c4559b97ffa749a658e7b" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.699689 12874 raft_consensus.cc:3032] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:07.699848 12874 raft_consensus.cc:729] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.700202 12874 consensus_queue.cc:260] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.700990 13004 raft_consensus.cc:504] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.701084 12874 raft_consensus.cc:3037] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.701654 13004 leader_election.cc:288] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:07.701779 12599 catalog_manager.cc:5484] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 3 to 4, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 4 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:07.702373 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b101e9e38b4048bc86d76f97201534" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.702559 13136 raft_consensus.cc:3037] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.703434 13004 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ddb529dd5534f8bb66562e8467c1f12"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:07.703608 13004 raft_consensus.cc:482] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.703701 13004 raft_consensus.cc:3037] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.705356 13136 raft_consensus.cc:2454] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 4.
I20240501 20:11:07.706651 13004 raft_consensus.cc:504] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.707265 13004 leader_election.cc:288] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 3 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:07.708307 13002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c3933feed404a4baee16428d2aafacf"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:07.708513 13002 raft_consensus.cc:482] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.708631 13002 raft_consensus.cc:3037] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.710973 13002 raft_consensus.cc:504] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.711210 12942 leader_election.cc:302] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:07.711414 12743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ddb529dd5534f8bb66562e8467c1f12" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.711516 15250 raft_consensus.cc:2781] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:11:07.711676 12743 raft_consensus.cc:3032] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.711762 12743 raft_consensus.cc:729] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.711750 13002 leader_election.cc:288] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:07.712065 12743 consensus_queue.cc:260] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.711758 15250 raft_consensus.cc:686] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:07.712374 12743 raft_consensus.cc:3037] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.712460 15250 consensus_queue.cc:237] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.713399 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c3933feed404a4baee16428d2aafacf" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.713622 13136 raft_consensus.cc:3037] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.702162 12877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b101e9e38b4048bc86d76f97201534" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.715279 12877 raft_consensus.cc:3032] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:07.715389 12877 raft_consensus.cc:729] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.715703 12877 consensus_queue.cc:260] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.715909 13136 raft_consensus.cc:2454] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 4.
I20240501 20:11:07.716104 12877 raft_consensus.cc:3037] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.716364 12942 leader_election.cc:302] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:07.716655 15250 raft_consensus.cc:2781] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:11:07.716804 15250 raft_consensus.cc:686] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:07.717072 15250 consensus_queue.cc:237] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.712536 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ddb529dd5534f8bb66562e8467c1f12" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.718842 12876 raft_consensus.cc:3037] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.712999 12875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c3933feed404a4baee16428d2aafacf" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.719117 12875 raft_consensus.cc:3032] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:07.719228 12875 raft_consensus.cc:729] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.719522 12875 consensus_queue.cc:260] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.719713 12877 raft_consensus.cc:2454] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 4.
I20240501 20:11:07.719877 12874 raft_consensus.cc:2454] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 4.
I20240501 20:11:07.719911 12875 raft_consensus.cc:3037] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.721120 12599 catalog_manager.cc:5484] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 3 to 4, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 4 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.721757 12599 catalog_manager.cc:5484] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 3 to 4, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 4 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.723521 13002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "436330ea0516443799906257122b38af"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:07.723649 12743 raft_consensus.cc:2396] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.723740 13002 raft_consensus.cc:482] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.723868 13002 raft_consensus.cc:3037] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.724988 12876 raft_consensus.cc:2454] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 3.
I20240501 20:11:07.722559 12875 raft_consensus.cc:2454] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 4.
I20240501 20:11:07.726444 13002 raft_consensus.cc:504] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.726761 12942 leader_election.cc:302] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:07.727020 15250 raft_consensus.cc:2781] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.727253 15250 raft_consensus.cc:686] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [term 3 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:07.727578 15250 consensus_queue.cc:237] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.728384 13002 leader_election.cc:288] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:07.729096 12743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436330ea0516443799906257122b38af" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.729331 12743 raft_consensus.cc:3032] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:07.729413 12743 raft_consensus.cc:729] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.729686 13002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dafd65d56bb243c78cb4490d7da7b18a"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:07.729867 13002 raft_consensus.cc:482] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.729794 12743 consensus_queue.cc:260] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.729985 13002 raft_consensus.cc:3037] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.730089 12743 raft_consensus.cc:3037] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.730183 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436330ea0516443799906257122b38af" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.730427 13136 raft_consensus.cc:3037] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.732844 12743 raft_consensus.cc:2396] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:07.732784 13002 raft_consensus.cc:504] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.733592 13002 leader_election.cc:288] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:07.734043 13136 raft_consensus.cc:2454] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 4.
I20240501 20:11:07.734200 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dafd65d56bb243c78cb4490d7da7b18a" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.734391 13134 raft_consensus.cc:3037] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.734496 13004 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "17276a219da74d95919c8f427c9920c5"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:07.734546 12942 leader_election.cc:302] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters: 4e9d799e45f8407590cb3096f5625c62
I20240501 20:11:07.734705 13004 raft_consensus.cc:482] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.734830 13004 raft_consensus.cc:3037] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.735066 12743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dafd65d56bb243c78cb4490d7da7b18a" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.735301 12743 raft_consensus.cc:3032] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:07.735409 12743 raft_consensus.cc:729] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.735734 12743 consensus_queue.cc:260] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.736269 13134 raft_consensus.cc:2454] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 4.
I20240501 20:11:07.736517 14590 raft_consensus.cc:2781] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:11:07.736640 12942 leader_election.cc:302] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:07.737607 13004 raft_consensus.cc:504] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.738368 13004 leader_election.cc:288] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:07.738471 12743 raft_consensus.cc:3037] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.739459 13002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "47b8c200bb05478db26c0d10b25a514c"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:07.739660 13002 raft_consensus.cc:482] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.739760 13002 raft_consensus.cc:3037] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.741007 12743 raft_consensus.cc:2396] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:07.743212 13002 raft_consensus.cc:504] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.744025 13002 leader_election.cc:288] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:07.745236 13004 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b96f18923f9d41878cb01202e77be033"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:07.745501 13004 raft_consensus.cc:482] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.745648 13004 raft_consensus.cc:3037] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.749264 13004 raft_consensus.cc:504] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.750083 13004 leader_election.cc:288] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 3 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:07.751269 13006 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4488bf7917574118915a9355219528ee"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:07.751464 13006 raft_consensus.cc:482] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.751598 13006 raft_consensus.cc:3037] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.752655 14590 raft_consensus.cc:686] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:07.752959 14590 consensus_queue.cc:237] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.754359 13006 raft_consensus.cc:504] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.754998 15254 raft_consensus.cc:2781] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:11:07.755198 15254 raft_consensus.cc:686] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:07.755276 13006 leader_election.cc:288] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 3 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:07.755654 15254 consensus_queue.cc:237] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.757077 13006 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "50561f1e0c5b47a78abb1fba84e725fa"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:07.757370 13006 raft_consensus.cc:482] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.757601 13006 raft_consensus.cc:3037] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.759902 12875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17276a219da74d95919c8f427c9920c5" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.760138 12875 raft_consensus.cc:3037] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.760172 12744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b8c200bb05478db26c0d10b25a514c" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.760376 12744 raft_consensus.cc:3032] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:07.760447 12744 raft_consensus.cc:729] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.760697 12744 consensus_queue.cc:260] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.761014 12744 raft_consensus.cc:3037] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.760980 13006 raft_consensus.cc:504] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.761536 13006 leader_election.cc:288] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:07.762506 13006 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1d207517fea64fe1a83a2409157ae2bd"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:07.762703 13006 raft_consensus.cc:482] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.762843 13006 raft_consensus.cc:3037] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.763003 12875 raft_consensus.cc:2396] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.764014 12744 raft_consensus.cc:2396] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.765120 12875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b8c200bb05478db26c0d10b25a514c" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.765339 12875 raft_consensus.cc:3037] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.765364 15049 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:07.765476 13006 raft_consensus.cc:504] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.765844 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96f18923f9d41878cb01202e77be033" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.766062 12876 raft_consensus.cc:3032] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.766142 12876 raft_consensus.cc:729] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.766266 13006 leader_election.cc:288] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 3 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:07.766453 12876 consensus_queue.cc:260] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.766788 12876 raft_consensus.cc:3037] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.767580 13006 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b6ba30e296834fb28bf4a88141c24826"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:07.767751 13006 raft_consensus.cc:482] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.767853 13006 raft_consensus.cc:3037] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.769351 12876 raft_consensus.cc:2396] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.770490 13006 raft_consensus.cc:504] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.771124 13006 leader_election.cc:288] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 3 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:07.771801 12744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4488bf7917574118915a9355219528ee" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.772033 12744 raft_consensus.cc:3032] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.772068 13006 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "70cd215b4f9445499e14c9fd18f7f7fe"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:07.772176 12744 raft_consensus.cc:729] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.772287 13006 raft_consensus.cc:482] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.772439 13006 raft_consensus.cc:3037] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.772512 12744 consensus_queue.cc:260] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.772583 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96f18923f9d41878cb01202e77be033" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.772780 13134 raft_consensus.cc:3037] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.772827 12744 raft_consensus.cc:3037] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.775348 13006 raft_consensus.cc:504] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.775615 13134 raft_consensus.cc:2454] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 3.
I20240501 20:11:07.776008 12744 raft_consensus.cc:2396] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.776172 13006 leader_election.cc:288] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:07.776867 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4488bf7917574118915a9355219528ee" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.777073 12876 raft_consensus.cc:3037] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.777511 13006 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5af8328d0e9248acb93e3707ffde7f8a"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:07.777707 13006 raft_consensus.cc:482] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.777825 13006 raft_consensus.cc:3037] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.778189 12875 raft_consensus.cc:2396] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.780123 12876 raft_consensus.cc:2454] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 3.
I20240501 20:11:07.780602 12743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17276a219da74d95919c8f427c9920c5" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.781847 12743 raft_consensus.cc:3032] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:07.781982 12743 raft_consensus.cc:729] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.781610 12744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50561f1e0c5b47a78abb1fba84e725fa" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.782307 12744 raft_consensus.cc:3032] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:07.782426 12744 raft_consensus.cc:729] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.782786 12744 consensus_queue.cc:260] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.783145 12744 raft_consensus.cc:3037] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.784057 12726 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:11:07.784718 13006 raft_consensus.cc:504] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.784844 12743 consensus_queue.cc:260] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.785162 12743 raft_consensus.cc:3037] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.785410 13006 leader_election.cc:288] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 3 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:07.785857 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50561f1e0c5b47a78abb1fba84e725fa" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.786079 13134 raft_consensus.cc:3037] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.786603 13006 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e1dbb57f13d348cc996b5248b7623b8e"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:07.786859 13006 raft_consensus.cc:482] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.787029 13006 raft_consensus.cc:3037] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.787163 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d207517fea64fe1a83a2409157ae2bd" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.787439 12876 raft_consensus.cc:3032] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.787578 12876 raft_consensus.cc:729] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.787907 12876 consensus_queue.cc:260] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.788349 12876 raft_consensus.cc:3037] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.788944 13134 raft_consensus.cc:2396] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.789697 13006 raft_consensus.cc:504] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.790367 13006 leader_election.cc:288] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 3 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:07.791313 12876 raft_consensus.cc:2454] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 3.
I20240501 20:11:07.791354 13004 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d85b0f46583649e9bee3f433ba6f928e"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:07.791522 13004 raft_consensus.cc:482] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.791638 13004 raft_consensus.cc:3037] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.792238 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d207517fea64fe1a83a2409157ae2bd" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.792421 13134 raft_consensus.cc:3037] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.792577 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ba30e296834fb28bf4a88141c24826" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.792759 12745 raft_consensus.cc:3032] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.792868 12745 raft_consensus.cc:729] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.793105 12745 consensus_queue.cc:260] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.793421 12745 raft_consensus.cc:3037] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.794205 13004 raft_consensus.cc:504] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.794912 13004 leader_election.cc:288] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 3 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:07.795099 13134 raft_consensus.cc:2454] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 3.
I20240501 20:11:07.795850 13002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e89e2dfe0abc487593a61f9f1b58ac65"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:07.796031 13002 raft_consensus.cc:482] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.796089 12745 raft_consensus.cc:2396] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.796154 13002 raft_consensus.cc:3037] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.799990 12942 leader_election.cc:302] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:07.799991 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70cd215b4f9445499e14c9fd18f7f7fe" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.800283 13134 raft_consensus.cc:3037] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.800522 15255 raft_consensus.cc:2781] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.800698 15255 raft_consensus.cc:686] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [term 3 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:07.800989 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ba30e296834fb28bf4a88141c24826" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.801005 15255 consensus_queue.cc:237] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.801445 12876 raft_consensus.cc:3037] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.798799 13002 raft_consensus.cc:504] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.803733 13002 leader_election.cc:288] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:07.803794 13134 raft_consensus.cc:2454] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 4.
I20240501 20:11:07.805007 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70cd215b4f9445499e14c9fd18f7f7fe" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.805320 12745 raft_consensus.cc:3032] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:07.805507 12745 raft_consensus.cc:729] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.805814 12745 consensus_queue.cc:260] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.806210 12745 raft_consensus.cc:3037] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.800705 12942 leader_election.cc:302] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:07.807109 15250 raft_consensus.cc:2781] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.807329 15250 raft_consensus.cc:686] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [term 3 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:07.807482 12942 leader_election.cc:302] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4720929b44c4560bfdf08107e137362; no voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c
I20240501 20:11:07.807736 15254 raft_consensus.cc:2726] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:11:07.807631 15250 consensus_queue.cc:237] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.807902 12942 leader_election.cc:302] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters: 4e9d799e45f8407590cb3096f5625c62
I20240501 20:11:07.808132 15254 raft_consensus.cc:2781] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.808346 15254 raft_consensus.cc:686] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [term 3 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:07.808612 15254 consensus_queue.cc:237] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.808750 13004 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f22d4331ed6f4e86b85a9919c219ab84"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:07.808984 13004 raft_consensus.cc:482] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.808955 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1dbb57f13d348cc996b5248b7623b8e" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.809090 13004 raft_consensus.cc:3037] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.809165 12942 leader_election.cc:302] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:07.809247 13136 raft_consensus.cc:2379] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 in current term 3: Already voted for candidate 4e9d799e45f8407590cb3096f5625c62 in this term.
I20240501 20:11:07.809446 14590 raft_consensus.cc:2781] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:11:07.810719 12599 catalog_manager.cc:5484] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 2 to 3, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 3 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.811909 13004 raft_consensus.cc:504] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.812409 12741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1dbb57f13d348cc996b5248b7623b8e" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.812639 12742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af8328d0e9248acb93e3707ffde7f8a" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.812589 12740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85b0f46583649e9bee3f433ba6f928e" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.812816 12741 raft_consensus.cc:2379] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 in current term 3: Already voted for candidate 4e9d799e45f8407590cb3096f5625c62 in this term.
I20240501 20:11:07.812839 12742 raft_consensus.cc:3032] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.812922 12742 raft_consensus.cc:729] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.813205 12742 consensus_queue.cc:260] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.813503 12742 raft_consensus.cc:3037] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.816238 12742 raft_consensus.cc:2396] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.817129 12739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89e2dfe0abc487593a61f9f1b58ac65" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.817387 12739 raft_consensus.cc:3032] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:07.817481 12739 raft_consensus.cc:729] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.817706 12739 consensus_queue.cc:260] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.818012 12739 raft_consensus.cc:3037] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.818291 13004 leader_election.cc:288] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:07.819546 13006 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "97dfa37b2ad2421b9242de5f0c227491"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:07.820091 13006 raft_consensus.cc:482] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.820603 13006 raft_consensus.cc:3037] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.820858 12739 raft_consensus.cc:2396] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.812894 12740 raft_consensus.cc:3032] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 20:11:07.821272 12740 raft_consensus.cc:729] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.821568 12740 consensus_queue.cc:260] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.821918 12740 raft_consensus.cc:3037] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.824479 13006 raft_consensus.cc:504] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.824641 12740 raft_consensus.cc:2396] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.825294 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22d4331ed6f4e86b85a9919c219ab84" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.825510 13136 raft_consensus.cc:3037] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.828336 13136 raft_consensus.cc:2454] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 4.
I20240501 20:11:07.829111 12875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85b0f46583649e9bee3f433ba6f928e" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.829313 12875 raft_consensus.cc:3037] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.830953 12740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22d4331ed6f4e86b85a9919c219ab84" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:07.831284 12740 raft_consensus.cc:3032] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:07.831450 12740 raft_consensus.cc:729] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:07.831825 12740 consensus_queue.cc:260] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.832286 12740 raft_consensus.cc:3037] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.832307 12875 raft_consensus.cc:2454] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 3.
I20240501 20:11:07.825044 12942 leader_election.cc:302] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4720929b44c4560bfdf08107e137362; no voters: 4e9d799e45f8407590cb3096f5625c62, bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:07.835333 12942 leader_election.cc:302] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:07.835976 12740 raft_consensus.cc:2396] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:07.836110 12942 leader_election.cc:302] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:07.836299 15250 raft_consensus.cc:2726] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 20:11:07.836539 15250 raft_consensus.cc:2781] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.836723 15250 raft_consensus.cc:686] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [term 3 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:07.837071 15250 consensus_queue.cc:237] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:07.837126 13136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dfa37b2ad2421b9242de5f0c227491" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.837318 13136 raft_consensus.cc:3037] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.838065 12745 raft_consensus.cc:2396] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:07.838815 12875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dfa37b2ad2421b9242de5f0c227491" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.839061 12875 raft_consensus.cc:3032] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:07.839150 12875 raft_consensus.cc:729] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:07.839354 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89e2dfe0abc487593a61f9f1b58ac65" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.839440 12875 consensus_queue.cc:260] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.839728 12874 raft_consensus.cc:3037] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.839819 12875 raft_consensus.cc:3037] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:07.843672 12874 raft_consensus.cc:2396] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.844219 12942 leader_election.cc:302] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4720929b44c4560bfdf08107e137362; no voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c
I20240501 20:11:07.844748 13136 raft_consensus.cc:2454] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 4.
I20240501 20:11:07.845405 12875 raft_consensus.cc:2454] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 4.
I20240501 20:11:07.845918 15255 raft_consensus.cc:2781] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:11:07.847941 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af8328d0e9248acb93e3707ffde7f8a" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:07.848241 13134 raft_consensus.cc:3037] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:11:07.851610 14786 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:07.855305 15266 raft_consensus.cc:2726] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:11:07.855341 13134 raft_consensus.cc:2454] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 3.
I20240501 20:11:07.856043 12942 leader_election.cc:302] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:07.856396 12942 leader_election.cc:302] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters: 4e9d799e45f8407590cb3096f5625c62
I20240501 20:11:07.857949 12744 raft_consensus.cc:2396] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.859213 12942 leader_election.cc:302] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4720929b44c4560bfdf08107e137362; no voters: 4e9d799e45f8407590cb3096f5625c62, bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:07.862068 12743 raft_consensus.cc:2396] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:07.862757 15266 raft_consensus.cc:2781] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:11:07.863036 15255 raft_consensus.cc:686] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:07.863204 15267 raft_consensus.cc:2781] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.863375 15255 consensus_queue.cc:237] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.865136 15255 raft_consensus.cc:2726] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:11:07.865578 12942 leader_election.cc:302] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4720929b44c4560bfdf08107e137362; no voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c
I20240501 20:11:07.835309 13006 leader_election.cc:288] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 4 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:07.867360 12876 raft_consensus.cc:2454] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 3.
I20240501 20:11:07.870127 12942 leader_election.cc:302] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters: 4e9d799e45f8407590cb3096f5625c62
I20240501 20:11:07.870885 15262 raft_consensus.cc:2726] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:11:07.871313 15262 raft_consensus.cc:2781] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:11:07.871616 15262 raft_consensus.cc:686] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [term 3 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:07.871114 14590 raft_consensus.cc:686] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:07.872654 14590 consensus_queue.cc:237] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.874578 15262 consensus_queue.cc:237] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.879093 13134 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eeba0c03d83a433d9a519ba9fe29b53a"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:07.879340 13134 raft_consensus.cc:482] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.879449 13134 raft_consensus.cc:3037] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:07.882417 13134 raft_consensus.cc:504] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.883507 13134 leader_election.cc:288] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:07.884096 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeba0c03d83a433d9a519ba9fe29b53a" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.884424 13006 raft_consensus.cc:3032] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:11:07.884593 13006 raft_consensus.cc:729] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:07.884980 13006 consensus_queue.cc:260] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.885464 13006 raft_consensus.cc:3037] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:07.886016 12599 catalog_manager.cc:5484] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 2 to 3, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 3 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:07.886663 12599 catalog_manager.cc:5484] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 2 to 3, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 3 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:07.887188 12599 catalog_manager.cc:5484] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 3 to 4, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 4 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.887661 12599 catalog_manager.cc:5484] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 3 to 4, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 4 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.888129 12599 catalog_manager.cc:5484] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 2 to 3, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 3 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.888599 12599 catalog_manager.cc:5484] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 3 to 4, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 4 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:07.889210 13006 raft_consensus.cc:2396] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:07.890733 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeba0c03d83a433d9a519ba9fe29b53a" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.891148 12876 raft_consensus.cc:3037] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:07.898901 12876 raft_consensus.cc:2454] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 5.
I20240501 20:11:07.899264 13134 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e077004855949878d554a2053131790"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:07.899679 13134 raft_consensus.cc:482] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.899861 13134 raft_consensus.cc:3037] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:07.902667 13072 leader_election.cc:302] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231; no voters: b4720929b44c4560bfdf08107e137362
I20240501 20:11:07.903143 15152 raft_consensus.cc:2781] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:11:07.903292 15152 raft_consensus.cc:686] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:07.903558 15152 consensus_queue.cc:237] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.907330 12856 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:11:07.907312 12599 catalog_manager.cc:5484] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 4 to 5, leader changed from b4720929b44c4560bfdf08107e137362 (127.29.219.67) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 5 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.907915 15267 raft_consensus.cc:686] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362 [term 3 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:07.908288 15267 consensus_queue.cc:237] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:07.909819 15266 raft_consensus.cc:686] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:07.910105 15266 consensus_queue.cc:237] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.906033 13134 raft_consensus.cc:504] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.914573 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e077004855949878d554a2053131790" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:07.914911 13006 raft_consensus.cc:3032] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:11:07.915279 13006 raft_consensus.cc:729] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:07.915570 13006 consensus_queue.cc:260] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.915899 13006 raft_consensus.cc:3037] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:07.919047 13006 raft_consensus.cc:2396] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:07.933082 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e077004855949878d554a2053131790" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.933534 12876 raft_consensus.cc:3037] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:07.936928 12876 raft_consensus.cc:2454] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 5.
I20240501 20:11:07.937278 13134 leader_election.cc:288] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:07.939960 14918 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:07.943014 13134 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e6a39cb39f7c4559b97ffa749a658e7b"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:07.943286 13134 raft_consensus.cc:482] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.943394 13134 raft_consensus.cc:3037] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:07.946290 13134 raft_consensus.cc:504] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:07.947767 13072 leader_election.cc:302] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231; no voters: b4720929b44c4560bfdf08107e137362
I20240501 20:11:07.948510 15140 raft_consensus.cc:2781] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:11:07.948704 15140 raft_consensus.cc:686] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:07.948948 15140 consensus_queue.cc:237] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:07.953143 12599 catalog_manager.cc:5484] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 4 to 5, leader changed from b4720929b44c4560bfdf08107e137362 (127.29.219.67) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 5 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:07.962692 12599 catalog_manager.cc:5484] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 2 to 3, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 3 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:07.963407 12599 catalog_manager.cc:5484] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 3 to 4, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 4 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:07.963922 12599 catalog_manager.cc:5484] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 2 to 3, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 3 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.964440 12599 catalog_manager.cc:5484] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 2 to 3, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 3 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.965014 12599 catalog_manager.cc:5484] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 3 to 4, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 4 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:07.971928 12986 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 20:11:07.994210 13134 leader_election.cc:288] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:07.994781 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a39cb39f7c4559b97ffa749a658e7b" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:07.995067 12876 raft_consensus.cc:3037] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:07.998137 12876 raft_consensus.cc:2454] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 5.
I20240501 20:11:07.998443 13004 raft_consensus.cc:1261] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Refusing update from remote peer 4e9d799e45f8407590cb3096f5625c62: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 20:11:07.998814 13134 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dafd65d56bb243c78cb4490d7da7b18a"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:07.999006 13134 raft_consensus.cc:482] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:07.999125 13134 raft_consensus.cc:3037] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.000097 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a39cb39f7c4559b97ffa749a658e7b" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.000285 13006 raft_consensus.cc:3032] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:11:08.000371 13006 raft_consensus.cc:729] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:08.000625 13006 consensus_queue.cc:260] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.000928 13006 raft_consensus.cc:3037] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.001477 15149 consensus_queue.cc:1035] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:08.001747 13134 raft_consensus.cc:504] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.002496 13134 leader_election.cc:288] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:08.002854 13004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dafd65d56bb243c78cb4490d7da7b18a" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.003065 13004 raft_consensus.cc:3032] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:11:08.003142 13004 raft_consensus.cc:729] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:08.003404 13004 consensus_queue.cc:260] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.003667 13004 raft_consensus.cc:3037] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.004005 12743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dafd65d56bb243c78cb4490d7da7b18a" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.004240 12743 raft_consensus.cc:3037] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.005324 13134 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "248dfbfcb6214244a410083e8de22743"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:08.005510 13134 raft_consensus.cc:482] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.006031 13134 raft_consensus.cc:3037] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.006501 13004 raft_consensus.cc:2396] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:08.008620 13134 raft_consensus.cc:504] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.008813 13006 raft_consensus.cc:2396] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.009290 13134 leader_election.cc:288] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 4 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:08.010756 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "248dfbfcb6214244a410083e8de22743" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.010967 12746 raft_consensus.cc:3037] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.015205 14654 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:08.015906 12746 raft_consensus.cc:2454] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 4.
I20240501 20:11:08.016376 13072 leader_election.cc:302] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:08.016809 15152 raft_consensus.cc:2781] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:11:08.016953 15152 raft_consensus.cc:686] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 4 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.017325 13136 raft_consensus.cc:1261] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Refusing update from remote peer 4e9d799e45f8407590cb3096f5625c62: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 20:11:08.017230 15152 consensus_queue.cc:237] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.017746 13134 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a8b101e9e38b4048bc86d76f97201534"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:08.017915 13134 raft_consensus.cc:482] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.018026 13134 raft_consensus.cc:3037] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.018759 13115 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 20:11:08.019308 15239 consensus_queue.cc:1035] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 20:11:08.020381 12604 catalog_manager.cc:5484] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 3 to 4, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 4 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.021380 13134 raft_consensus.cc:504] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.023080 13134 leader_election.cc:288] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:08.025408 13072 leader_election.cc:302] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231; no voters: b4720929b44c4560bfdf08107e137362
I20240501 20:11:08.025673 15140 raft_consensus.cc:2781] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:11:08.025796 15140 raft_consensus.cc:686] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.026046 15140 consensus_queue.cc:237] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.026552 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b101e9e38b4048bc86d76f97201534" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.026799 12874 raft_consensus.cc:3037] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.028291 13136 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c3933feed404a4baee16428d2aafacf"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:08.028445 13136 raft_consensus.cc:482] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.028537 13136 raft_consensus.cc:3037] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.029788 12874 raft_consensus.cc:2454] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 5.
I20240501 20:11:08.030357 13072 leader_election.cc:302] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:08.030649 15152 raft_consensus.cc:2781] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:11:08.030794 15152 raft_consensus.cc:686] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.030974 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b101e9e38b4048bc86d76f97201534" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.031036 15152 consensus_queue.cc:237] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.031199 13002 raft_consensus.cc:3032] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:11:08.031268 13002 raft_consensus.cc:729] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:08.031527 13002 consensus_queue.cc:260] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.031818 13002 raft_consensus.cc:3037] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.032954 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "248dfbfcb6214244a410083e8de22743" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.033139 12876 raft_consensus.cc:3032] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:08.033247 12876 raft_consensus.cc:729] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.033483 12876 consensus_queue.cc:260] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.033854 12876 raft_consensus.cc:3037] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.036692 12876 raft_consensus.cc:2454] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 4.
I20240501 20:11:08.038378 12601 catalog_manager.cc:5484] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 4 to 5, leader changed from b4720929b44c4560bfdf08107e137362 (127.29.219.67) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 5 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:08.039096 12743 raft_consensus.cc:2396] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:08.039008 12601 catalog_manager.cc:5484] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 4 to 5, leader changed from b4720929b44c4560bfdf08107e137362 (127.29.219.67) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 5 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.039723 13072 leader_election.cc:302] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb1be87a7aee48a4b405b9d209c61231; no voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362
I20240501 20:11:08.040174 15123 raft_consensus.cc:2726] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240501 20:11:08.041873 13136 raft_consensus.cc:504] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.045265 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c3933feed404a4baee16428d2aafacf" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.046386 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c3933feed404a4baee16428d2aafacf" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.046640 12876 raft_consensus.cc:3037] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.048206 13006 raft_consensus.cc:3032] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:11:08.048398 13136 leader_election.cc:288] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:08.048552 13006 raft_consensus.cc:729] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:08.049278 13006 consensus_queue.cc:260] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.049376 12876 raft_consensus.cc:2454] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 5.
I20240501 20:11:08.049628 13002 raft_consensus.cc:2396] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.049782 13072 leader_election.cc:302] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:08.050058 13006 raft_consensus.cc:3037] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.050169 15123 raft_consensus.cc:2781] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:11:08.050329 15123 raft_consensus.cc:686] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.050586 15123 consensus_queue.cc:237] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.053515 13006 raft_consensus.cc:2396] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.055320 13136 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "436330ea0516443799906257122b38af"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:08.055523 13136 raft_consensus.cc:482] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.055640 13136 raft_consensus.cc:3037] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.057202 12604 catalog_manager.cc:5484] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 4 to 5, leader changed from b4720929b44c4560bfdf08107e137362 (127.29.219.67) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 5 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:08.062069 13136 raft_consensus.cc:504] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.063233 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436330ea0516443799906257122b38af" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.063431 13002 raft_consensus.cc:3032] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:11:08.063504 13002 raft_consensus.cc:729] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:08.063781 13002 consensus_queue.cc:260] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.064042 13002 raft_consensus.cc:3037] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.066919 13002 raft_consensus.cc:2396] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:08.070078 12743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436330ea0516443799906257122b38af" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.070343 12743 raft_consensus.cc:3037] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.073954 12743 raft_consensus.cc:2396] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:08.074733 13072 leader_election.cc:302] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb1be87a7aee48a4b405b9d209c61231; no voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362
I20240501 20:11:08.075143 15152 raft_consensus.cc:2726] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240501 20:11:08.074714 13136 leader_election.cc:288] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:08.079872 13136 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5f728b714ca142648ba5aa3459f7ae80"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:08.080118 13136 raft_consensus.cc:482] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.080258 13136 raft_consensus.cc:3037] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.083236 13136 raft_consensus.cc:504] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.083987 13136 leader_election.cc:288] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:08.084723 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f728b714ca142648ba5aa3459f7ae80" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.084968 12876 raft_consensus.cc:3037] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.084957 13134 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f22d4331ed6f4e86b85a9919c219ab84"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:08.085155 13134 raft_consensus.cc:482] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.085269 13134 raft_consensus.cc:3037] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.087971 12876 raft_consensus.cc:2396] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:08.087894 13134 raft_consensus.cc:504] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.088910 13134 leader_election.cc:288] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:08.089011 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22d4331ed6f4e86b85a9919c219ab84" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.089246 12746 raft_consensus.cc:3037] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.089910 13134 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "97dfa37b2ad2421b9242de5f0c227491"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:08.090083 13134 raft_consensus.cc:482] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.090205 13134 raft_consensus.cc:3037] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.091221 12743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f728b714ca142648ba5aa3459f7ae80" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.091212 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22d4331ed6f4e86b85a9919c219ab84" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.091429 12743 raft_consensus.cc:3037] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.091440 13002 raft_consensus.cc:3032] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:11:08.091516 13002 raft_consensus.cc:729] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:08.091790 13002 consensus_queue.cc:260] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.092142 13002 raft_consensus.cc:3037] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.093313 13134 raft_consensus.cc:504] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.094259 12743 raft_consensus.cc:2454] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 5.
I20240501 20:11:08.094529 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dfa37b2ad2421b9242de5f0c227491" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.094708 13134 leader_election.cc:288] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:08.094760 12874 raft_consensus.cc:3037] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.094879 13072 leader_election.cc:302] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, bb1be87a7aee48a4b405b9d209c61231; no voters: 77bf705599644326b6e71b5caf32fc7c
I20240501 20:11:08.095299 15152 raft_consensus.cc:2781] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:11:08.095439 15152 raft_consensus.cc:686] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.095696 15152 consensus_queue.cc:237] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.097288 13134 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e00b9b3645a04a4686e5759bd8201eca"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:08.097499 13134 raft_consensus.cc:482] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.097592 13134 raft_consensus.cc:3037] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.097771 12874 raft_consensus.cc:2454] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 5.
I20240501 20:11:08.098670 13072 leader_election.cc:302] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:08.098970 15123 raft_consensus.cc:2781] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:11:08.099149 15123 raft_consensus.cc:686] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.099399 15123 consensus_queue.cc:237] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.099618 13002 raft_consensus.cc:2396] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:08.100217 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dfa37b2ad2421b9242de5f0c227491" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.100340 13134 raft_consensus.cc:504] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.100492 13006 raft_consensus.cc:3032] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:11:08.100582 13006 raft_consensus.cc:729] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:08.100925 13134 leader_election.cc:288] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:08.100858 13006 consensus_queue.cc:260] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.101162 13006 raft_consensus.cc:3037] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.102109 12746 raft_consensus.cc:2396] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:08.102406 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00b9b3645a04a4686e5759bd8201eca" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.102272 12744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00b9b3645a04a4686e5759bd8201eca" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.102720 12744 raft_consensus.cc:3037] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.103235 13072 leader_election.cc:302] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb1be87a7aee48a4b405b9d209c61231; no voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362
I20240501 20:11:08.103701 15140 raft_consensus.cc:2726] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240501 20:11:08.103895 12876 raft_consensus.cc:3037] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.105579 12744 raft_consensus.cc:2454] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 5.
I20240501 20:11:08.106683 13072 leader_election.cc:302] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:08.107034 15140 raft_consensus.cc:2781] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:11:08.107244 15140 raft_consensus.cc:686] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.107625 15140 consensus_queue.cc:237] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.110824 12876 raft_consensus.cc:2396] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:08.112314 13134 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "66c3dd1267b741d4a5001ac7ae1d57ca"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:08.112538 13134 raft_consensus.cc:482] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.112725 13134 raft_consensus.cc:3037] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.114475 13006 raft_consensus.cc:2396] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.115806 13134 raft_consensus.cc:504] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.116609 13134 leader_election.cc:288] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:08.117203 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c3dd1267b741d4a5001ac7ae1d57ca" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.117229 12744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c3dd1267b741d4a5001ac7ae1d57ca" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.117523 12876 raft_consensus.cc:3037] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.117524 12744 raft_consensus.cc:3037] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.117667 13136 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "50561f1e0c5b47a78abb1fba84e725fa"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:08.117879 13136 raft_consensus.cc:482] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.117992 13136 raft_consensus.cc:3037] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.120491 12744 raft_consensus.cc:2454] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 5.
I20240501 20:11:08.120877 13136 raft_consensus.cc:504] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.121286 13072 leader_election.cc:302] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:08.121596 15140 raft_consensus.cc:2781] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:11:08.121706 13136 leader_election.cc:288] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:08.122068 15140 raft_consensus.cc:686] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.122308 12743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50561f1e0c5b47a78abb1fba84e725fa" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.122525 12743 raft_consensus.cc:3037] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.122534 15140 consensus_queue.cc:237] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.122752 13137 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1d207517fea64fe1a83a2409157ae2bd"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:08.123045 13137 raft_consensus.cc:482] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.123195 13137 raft_consensus.cc:3037] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.125327 12743 raft_consensus.cc:2396] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:08.126055 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50561f1e0c5b47a78abb1fba84e725fa" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.126255 13006 raft_consensus.cc:3037] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.126199 13137 raft_consensus.cc:504] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.127506 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d207517fea64fe1a83a2409157ae2bd" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.127717 12874 raft_consensus.cc:3037] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.129659 13006 raft_consensus.cc:2454] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 5.
I20240501 20:11:08.130007 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d207517fea64fe1a83a2409157ae2bd" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.130120 13071 leader_election.cc:302] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters: 4e9d799e45f8407590cb3096f5625c62
I20240501 20:11:08.130209 13006 raft_consensus.cc:3032] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:08.130276 13006 raft_consensus.cc:729] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:08.130573 13006 consensus_queue.cc:260] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.130875 13006 raft_consensus.cc:3037] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.132093 15123 raft_consensus.cc:2781] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:11:08.134236 13006 raft_consensus.cc:2396] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:08.134595 13137 leader_election.cc:288] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 4 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:08.135962 12874 raft_consensus.cc:2454] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 4.
I20240501 20:11:08.136893 13072 leader_election.cc:302] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231; no voters: b4720929b44c4560bfdf08107e137362
I20240501 20:11:08.141455 13134 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "70cd215b4f9445499e14c9fd18f7f7fe"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:08.141796 13134 raft_consensus.cc:482] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.141954 13134 raft_consensus.cc:3037] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.145867 13134 raft_consensus.cc:504] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.147473 12743 raft_consensus.cc:1261] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Refusing update from remote peer b4720929b44c4560bfdf08107e137362: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 20:11:08.147464 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70cd215b4f9445499e14c9fd18f7f7fe" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.147647 12743 pending_rounds.cc:76] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62: Aborting all ops after (but not including) 3
I20240501 20:11:08.147764 13006 raft_consensus.cc:3032] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:11:08.147876 13006 raft_consensus.cc:729] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:08.148134 13134 leader_election.cc:288] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:08.148108 12743 pending_rounds.cc:97] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 20:11:08.148543 12743 raft_consensus.cc:2866] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:11:08.148561 13006 consensus_queue.cc:260] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.149116 13006 raft_consensus.cc:3037] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.149343 13134 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a04bef0a392e42ebae58a89d46e2f373"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:08.149542 13134 raft_consensus.cc:482] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.149462 15267 consensus_queue.cc:1035] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:08.149662 13134 raft_consensus.cc:3037] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.152581 13006 raft_consensus.cc:2396] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:08.153431 13134 raft_consensus.cc:504] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.154104 13134 leader_election.cc:288] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:08.155272 13137 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9a992a1cf4484b298200677ee8b56424"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:08.155544 13137 raft_consensus.cc:482] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.155704 13137 raft_consensus.cc:3037] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.158917 12874 raft_consensus.cc:1261] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Refusing update from remote peer b4720929b44c4560bfdf08107e137362: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 20:11:08.159863 15267 consensus_queue.cc:1035] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:08.161828 13137 raft_consensus.cc:504] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.162798 13137 leader_election.cc:288] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:08.164315 13137 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b96f18923f9d41878cb01202e77be033"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:08.164621 13137 raft_consensus.cc:482] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.164853 13137 raft_consensus.cc:3037] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.168823 13137 raft_consensus.cc:504] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.170599 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96f18923f9d41878cb01202e77be033" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.171028 13006 raft_consensus.cc:3032] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:08.171193 13006 raft_consensus.cc:729] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:08.171645 13006 consensus_queue.cc:260] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.172351 13006 raft_consensus.cc:3037] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.172564 15152 raft_consensus.cc:2781] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:11:08.172799 15152 raft_consensus.cc:686] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 4 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.173187 15152 consensus_queue.cc:237] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.175397 12876 raft_consensus.cc:2396] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:08.176985 13006 raft_consensus.cc:2396] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:08.177726 13137 leader_election.cc:288] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 4 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:08.179085 13137 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5af8328d0e9248acb93e3707ffde7f8a"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:08.179289 13137 raft_consensus.cc:482] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.179380 13137 raft_consensus.cc:3037] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.183200 13137 raft_consensus.cc:504] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.184839 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af8328d0e9248acb93e3707ffde7f8a" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.185211 13006 raft_consensus.cc:3032] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:08.185374 13006 raft_consensus.cc:729] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:08.185915 13006 consensus_queue.cc:260] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.186406 13006 raft_consensus.cc:3037] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.190582 13006 raft_consensus.cc:2396] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:08.191009 13137 leader_election.cc:288] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 4 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:08.191430 15123 raft_consensus.cc:686] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.192698 15123 consensus_queue.cc:237] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.192821 13137 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e1dbb57f13d348cc996b5248b7623b8e"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:08.193885 13137 raft_consensus.cc:482] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.194124 13137 raft_consensus.cc:3037] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.198978 13137 raft_consensus.cc:504] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.200542 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1dbb57f13d348cc996b5248b7623b8e" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.200904 13006 raft_consensus.cc:3037] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.205004 13006 raft_consensus.cc:2454] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 4.
I20240501 20:11:08.205644 13071 leader_election.cc:302] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:08.206045 15152 raft_consensus.cc:2781] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:11:08.206252 15152 raft_consensus.cc:686] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 4 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.206573 15152 consensus_queue.cc:237] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
W20240501 20:11:08.206658 13072 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.219.66:32867, user_credentials={real_user=slave}} blocked reactor thread for 69745.8us
I20240501 20:11:08.207403 13137 leader_election.cc:288] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 4 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:08.207588 12743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70cd215b4f9445499e14c9fd18f7f7fe" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.207896 12743 raft_consensus.cc:3037] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.208954 13137 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3eac603886e4a84877d71ccebd0dff0"
dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
from {username='slave'} at 127.0.0.1:56532
I20240501 20:11:08.209149 13137 raft_consensus.cc:482] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.209254 13137 raft_consensus.cc:3037] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.211460 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96f18923f9d41878cb01202e77be033" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.211603 13137 raft_consensus.cc:504] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.211750 12874 raft_consensus.cc:3037] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.211745 12604 catalog_manager.cc:5484] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 3 to 5, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 5 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.212210 13137 leader_election.cc:288] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:08.214484 12744 raft_consensus.cc:1261] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Refusing update from remote peer b4720929b44c4560bfdf08107e137362: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 20:11:08.214658 12744 pending_rounds.cc:76] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62: Aborting all ops after (but not including) 3
I20240501 20:11:08.214732 12744 pending_rounds.cc:97] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 20:11:08.214711 12742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3eac603886e4a84877d71ccebd0dff0" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.214826 12744 raft_consensus.cc:2866] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:11:08.214936 12742 raft_consensus.cc:3037] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.215024 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04bef0a392e42ebae58a89d46e2f373" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.215270 12746 raft_consensus.cc:3037] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.215451 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a992a1cf4484b298200677ee8b56424" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.215657 12745 raft_consensus.cc:3037] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.218086 12742 raft_consensus.cc:2454] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 5.
I20240501 20:11:08.218475 12745 raft_consensus.cc:2454] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 5.
I20240501 20:11:08.218792 12740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af8328d0e9248acb93e3707ffde7f8a" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.218959 12740 raft_consensus.cc:3037] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.220610 12874 raft_consensus.cc:2396] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:08.221740 12740 raft_consensus.cc:2396] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:08.221801 12876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a992a1cf4484b298200677ee8b56424" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.222036 12876 raft_consensus.cc:3037] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.222052 12739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1dbb57f13d348cc996b5248b7623b8e" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.222235 12739 raft_consensus.cc:3032] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:08.222334 12739 raft_consensus.cc:729] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.222594 12739 consensus_queue.cc:260] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.222993 12739 raft_consensus.cc:3037] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.223870 12743 raft_consensus.cc:2396] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:08.224205 12875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04bef0a392e42ebae58a89d46e2f373" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.224411 12875 raft_consensus.cc:3037] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.225739 12739 raft_consensus.cc:2454] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 4.
I20240501 20:11:08.226011 12876 raft_consensus.cc:2396] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:08.226963 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3eac603886e4a84877d71ccebd0dff0" candidate_uuid: "bb1be87a7aee48a4b405b9d209c61231" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.227173 12874 raft_consensus.cc:3037] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.227509 12746 raft_consensus.cc:2454] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb1be87a7aee48a4b405b9d209c61231 in term 5.
I20240501 20:11:08.228288 12875 raft_consensus.cc:2396] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:08.228806 13072 leader_election.cc:302] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb1be87a7aee48a4b405b9d209c61231; no voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362
I20240501 20:11:08.229483 15123 raft_consensus.cc:2726] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:11:08.230252 12874 raft_consensus.cc:2396] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bb1be87a7aee48a4b405b9d209c61231 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:08.230729 15267 consensus_queue.cc:1035] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:08.231145 13072 leader_election.cc:302] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:08.231505 13072 leader_election.cc:302] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb1be87a7aee48a4b405b9d209c61231; no voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362
I20240501 20:11:08.231829 15152 raft_consensus.cc:2781] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:11:08.232000 15152 raft_consensus.cc:686] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.231971 13072 leader_election.cc:302] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb1be87a7aee48a4b405b9d209c61231; no voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362
I20240501 20:11:08.232285 15152 consensus_queue.cc:237] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.232494 13072 leader_election.cc:302] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, bb1be87a7aee48a4b405b9d209c61231; no voters: 77bf705599644326b6e71b5caf32fc7c
I20240501 20:11:08.232841 13072 leader_election.cc:302] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, bb1be87a7aee48a4b405b9d209c61231; no voters: 77bf705599644326b6e71b5caf32fc7c
I20240501 20:11:08.233199 15123 raft_consensus.cc:2726] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 20:11:08.233402 15123 raft_consensus.cc:2726] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240501 20:11:08.233530 15123 raft_consensus.cc:2781] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:11:08.233640 15123 raft_consensus.cc:686] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.233413 12604 catalog_manager.cc:5484] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 3 to 4, leader changed from b4720929b44c4560bfdf08107e137362 (127.29.219.67) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 4 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.233918 15123 consensus_queue.cc:237] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.234162 12604 catalog_manager.cc:5484] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 4 to 5, leader changed from b4720929b44c4560bfdf08107e137362 (127.29.219.67) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 5 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.235751 15140 raft_consensus.cc:2781] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:11:08.235913 15140 raft_consensus.cc:686] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Leader. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.236207 15140 consensus_queue.cc:237] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.237020 12604 catalog_manager.cc:5484] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 3 to 5, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 5 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.237676 12604 catalog_manager.cc:5484] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 3 to 5, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 5 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.238251 12604 catalog_manager.cc:5484] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 3 to 4, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 4 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:08.243054 12876 raft_consensus.cc:1261] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Refusing update from remote peer b4720929b44c4560bfdf08107e137362: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 20:11:08.242242 12604 catalog_manager.cc:5484] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 3 to 5, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 5 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:08.243324 12875 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e077004855949878d554a2053131790"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:08.249531 12875 raft_consensus.cc:482] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.249696 12875 raft_consensus.cc:3037] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.251106 15268 consensus_queue.cc:1035] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:08.252728 12875 raft_consensus.cc:504] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.253583 12604 catalog_manager.cc:5484] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 3 to 5, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 5 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.254117 12875 leader_election.cc:288] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 6 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:08.254187 12604 catalog_manager.cc:5484] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 3 to 5, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 5 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.254719 12604 catalog_manager.cc:5484] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 reported cstate change: term changed from 3 to 5, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68). New cstate: current_term: 5 leader_uuid: "bb1be87a7aee48a4b405b9d209c61231" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.255026 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e077004855949878d554a2053131790" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.255226 13006 raft_consensus.cc:3037] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.255432 12876 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eeba0c03d83a433d9a519ba9fe29b53a"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:08.255658 12876 raft_consensus.cc:482] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.255827 12876 raft_consensus.cc:3037] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.258044 13006 raft_consensus.cc:2396] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.259655 12876 raft_consensus.cc:504] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.261272 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e077004855949878d554a2053131790" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.261467 12876 leader_election.cc:288] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 6 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:08.261512 13137 raft_consensus.cc:3032] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:11:08.261624 13137 raft_consensus.cc:729] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.261927 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeba0c03d83a433d9a519ba9fe29b53a" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.261965 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeba0c03d83a433d9a519ba9fe29b53a" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.262140 13134 raft_consensus.cc:3032] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:11:08.262166 13006 raft_consensus.cc:3037] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.262217 13134 raft_consensus.cc:729] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.262499 13134 consensus_queue.cc:260] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.262826 13134 raft_consensus.cc:3037] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.262852 12875 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e6a39cb39f7c4559b97ffa749a658e7b"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:08.263114 12875 raft_consensus.cc:482] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.263238 12875 raft_consensus.cc:3037] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.265201 13006 raft_consensus.cc:2396] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.265686 13134 raft_consensus.cc:2396] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.266297 13137 consensus_queue.cc:260] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.266638 13137 raft_consensus.cc:3037] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.267120 12875 raft_consensus.cc:504] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.268362 12875 leader_election.cc:288] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 6 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:08.268865 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a39cb39f7c4559b97ffa749a658e7b" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.269070 13006 raft_consensus.cc:3037] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.269443 13137 raft_consensus.cc:2396] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.269722 12876 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3eac603886e4a84877d71ccebd0dff0"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:08.270151 12812 leader_election.cc:302] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c; no voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:08.270491 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a39cb39f7c4559b97ffa749a658e7b" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.270769 13137 raft_consensus.cc:3032] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:11:08.270851 12812 leader_election.cc:302] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c; no voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:08.271100 13137 raft_consensus.cc:729] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.271375 15165 raft_consensus.cc:2726] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 20:11:08.271548 15165 raft_consensus.cc:2726] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 20:11:08.271533 13137 consensus_queue.cc:260] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.271912 13137 raft_consensus.cc:3037] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.271966 13006 raft_consensus.cc:2396] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.272002 12876 raft_consensus.cc:482] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.272372 12876 raft_consensus.cc:3037] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.276414 12876 raft_consensus.cc:504] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.277510 13137 raft_consensus.cc:2396] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.278146 12812 leader_election.cc:302] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c; no voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:08.278512 15013 raft_consensus.cc:2726] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 20:11:08.279654 12876 leader_election.cc:288] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 6 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:08.280236 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3eac603886e4a84877d71ccebd0dff0" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.280472 12745 raft_consensus.cc:3037] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.280773 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3eac603886e4a84877d71ccebd0dff0" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.280995 13137 raft_consensus.cc:3032] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:11:08.281080 13137 raft_consensus.cc:729] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.281363 13137 consensus_queue.cc:260] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.281641 13137 raft_consensus.cc:3037] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.283313 12745 raft_consensus.cc:2454] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 6.
I20240501 20:11:08.283289 12874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ddb529dd5534f8bb66562e8467c1f12"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:08.283536 12874 raft_consensus.cc:482] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.283684 12874 raft_consensus.cc:3037] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.283995 12812 leader_election.cc:302] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters:
I20240501 20:11:08.284417 15013 raft_consensus.cc:2781] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:11:08.284592 15013 raft_consensus.cc:686] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 6 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.284888 15013 consensus_queue.cc:237] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.286084 13137 raft_consensus.cc:2396] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.288134 12874 raft_consensus.cc:504] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.288975 12874 leader_election.cc:288] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 4 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:08.289688 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ddb529dd5534f8bb66562e8467c1f12" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.289995 13006 raft_consensus.cc:3032] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:08.290089 13006 raft_consensus.cc:729] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:08.290465 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ddb529dd5534f8bb66562e8467c1f12" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.290459 13006 consensus_queue.cc:260] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.291275 13006 raft_consensus.cc:3037] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.290844 12745 raft_consensus.cc:3037] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.294680 12745 raft_consensus.cc:2454] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 4.
I20240501 20:11:08.294939 13006 raft_consensus.cc:2454] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 4.
I20240501 20:11:08.295536 12811 leader_election.cc:302] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:08.295799 12874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c3933feed404a4baee16428d2aafacf"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:08.296005 12874 raft_consensus.cc:482] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.296157 12874 raft_consensus.cc:3037] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.296257 15013 raft_consensus.cc:2781] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:11:08.296063 12604 catalog_manager.cc:5484] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 5 to 6, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 6 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:08.296567 15013 raft_consensus.cc:686] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 4 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.296854 15013 consensus_queue.cc:237] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.299022 12874 raft_consensus.cc:504] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.299790 12874 leader_election.cc:288] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 6 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:08.300395 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c3933feed404a4baee16428d2aafacf" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.300704 13006 raft_consensus.cc:3037] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.300957 12874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a8b101e9e38b4048bc86d76f97201534"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:08.301159 12874 raft_consensus.cc:482] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.301265 12874 raft_consensus.cc:3037] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.302057 12601 catalog_manager.cc:5484] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 3 to 4, leader changed from b4720929b44c4560bfdf08107e137362 (127.29.219.67) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 4 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:08.303910 12874 raft_consensus.cc:504] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.304908 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b101e9e38b4048bc86d76f97201534" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.305143 13002 raft_consensus.cc:3037] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.305943 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c3933feed404a4baee16428d2aafacf" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.306627 13137 raft_consensus.cc:3032] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:11:08.306816 13137 raft_consensus.cc:729] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.307108 13137 consensus_queue.cc:260] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.307516 13137 raft_consensus.cc:3037] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.307790 12874 leader_election.cc:288] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 6 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:08.307852 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b101e9e38b4048bc86d76f97201534" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.308033 13134 raft_consensus.cc:3032] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:11:08.308125 13134 raft_consensus.cc:729] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.308357 13134 consensus_queue.cc:260] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.308634 13134 raft_consensus.cc:3037] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.308925 12874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5f728b714ca142648ba5aa3459f7ae80"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:08.309093 12874 raft_consensus.cc:482] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.309216 12874 raft_consensus.cc:3037] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.311619 13134 raft_consensus.cc:2396] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.311924 12874 raft_consensus.cc:504] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.312450 13002 raft_consensus.cc:2396] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.312759 12874 leader_election.cc:288] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 6 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:08.312988 12811 leader_election.cc:302] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c; no voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:08.313532 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f728b714ca142648ba5aa3459f7ae80" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.313521 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f728b714ca142648ba5aa3459f7ae80" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.313764 12745 raft_consensus.cc:3037] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.313807 13134 raft_consensus.cc:3032] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:11:08.313970 13134 raft_consensus.cc:729] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.314288 13134 consensus_queue.cc:260] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.314603 13134 raft_consensus.cc:3037] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.316593 12745 raft_consensus.cc:2454] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 6.
I20240501 20:11:08.317433 12812 leader_election.cc:302] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters:
I20240501 20:11:08.317765 15165 raft_consensus.cc:2726] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 20:11:08.317818 13134 raft_consensus.cc:2396] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.317881 12874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "97dfa37b2ad2421b9242de5f0c227491"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:08.317955 15165 raft_consensus.cc:2781] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:11:08.318082 12874 raft_consensus.cc:482] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.318114 15165 raft_consensus.cc:686] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 6 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.318215 12874 raft_consensus.cc:3037] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.318408 15165 consensus_queue.cc:237] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.320802 13137 raft_consensus.cc:2396] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.321043 12874 raft_consensus.cc:504] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.322046 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dfa37b2ad2421b9242de5f0c227491" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.322108 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97dfa37b2ad2421b9242de5f0c227491" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.322312 13002 raft_consensus.cc:3037] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.322328 13137 raft_consensus.cc:3032] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:11:08.322505 13137 raft_consensus.cc:729] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.322865 13137 consensus_queue.cc:260] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.323237 13137 raft_consensus.cc:3037] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.323709 13006 raft_consensus.cc:2396] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.324295 12811 leader_election.cc:302] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c; no voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:08.325963 15165 raft_consensus.cc:2726] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 20:11:08.326323 12874 leader_election.cc:288] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 6 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:08.326436 13002 raft_consensus.cc:2396] T 97dfa37b2ad2421b9242de5f0c227491 P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.327709 12604 catalog_manager.cc:5484] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 5 to 6, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 6 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.329206 12874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c1211cfd40674438a14a49818863b3f0"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:08.329438 12874 raft_consensus.cc:482] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.330158 12874 raft_consensus.cc:3037] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.330471 13137 raft_consensus.cc:2396] T 97dfa37b2ad2421b9242de5f0c227491 P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.331552 12812 leader_election.cc:302] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c; no voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:08.332525 15013 raft_consensus.cc:2726] T 97dfa37b2ad2421b9242de5f0c227491 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 20:11:08.332868 12874 raft_consensus.cc:504] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.333832 12874 leader_election.cc:288] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 5 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:08.333973 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1211cfd40674438a14a49818863b3f0" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.334228 13002 raft_consensus.cc:3037] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.334858 12874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e00b9b3645a04a4686e5759bd8201eca"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:08.334861 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1211cfd40674438a14a49818863b3f0" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.335073 12745 raft_consensus.cc:3037] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.335143 12874 raft_consensus.cc:482] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.335335 12874 raft_consensus.cc:3037] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.337213 13002 raft_consensus.cc:2454] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 5.
I20240501 20:11:08.337644 12811 leader_election.cc:302] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:08.337953 12745 raft_consensus.cc:2396] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:08.338002 15013 raft_consensus.cc:2781] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:11:08.338172 15013 raft_consensus.cc:686] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 5 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.338450 15013 consensus_queue.cc:237] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.340188 12874 raft_consensus.cc:504] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.341069 12745 raft_consensus.cc:1261] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Refusing update from remote peer b4720929b44c4560bfdf08107e137362: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 20:11:08.341188 12745 pending_rounds.cc:76] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62: Aborting all ops after (but not including) 3
I20240501 20:11:08.341246 12745 pending_rounds.cc:97] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 20:11:08.341328 12745 raft_consensus.cc:2866] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:11:08.341473 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00b9b3645a04a4686e5759bd8201eca" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.341699 13137 raft_consensus.cc:3032] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:11:08.341828 13137 raft_consensus.cc:729] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.342134 13137 consensus_queue.cc:260] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.342476 13137 raft_consensus.cc:3037] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.342478 15268 consensus_queue.cc:1035] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:08.344672 12874 leader_election.cc:288] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 6 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:08.345813 13137 raft_consensus.cc:2396] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.349282 12604 catalog_manager.cc:5484] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 3 to 5, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 5 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:08.349547 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00b9b3645a04a4686e5759bd8201eca" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.349752 12745 raft_consensus.cc:3037] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.350317 12876 raft_consensus.cc:1261] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Refusing update from remote peer b4720929b44c4560bfdf08107e137362: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 20:11:08.353209 12745 raft_consensus.cc:2454] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 6.
I20240501 20:11:08.353593 12812 leader_election.cc:302] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters: bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:08.354075 15013 raft_consensus.cc:2781] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:11:08.354225 15013 raft_consensus.cc:686] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 6 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.354506 15013 consensus_queue.cc:237] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.354959 15267 consensus_queue.cc:1035] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:08.355968 12746 raft_consensus.cc:1261] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Refusing update from remote peer b4720929b44c4560bfdf08107e137362: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 20:11:08.356101 12746 pending_rounds.cc:76] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62: Aborting all ops after (but not including) 3
I20240501 20:11:08.356161 12746 pending_rounds.cc:97] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 20:11:08.356256 12746 raft_consensus.cc:2866] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 20:11:08.357095 15250 consensus_queue.cc:1035] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:08.361261 12874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "66c3dd1267b741d4a5001ac7ae1d57ca"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:08.361588 12874 raft_consensus.cc:482] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.361780 12874 raft_consensus.cc:3037] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.362953 12875 raft_consensus.cc:1261] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Refusing update from remote peer b4720929b44c4560bfdf08107e137362: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 20:11:08.365584 12874 raft_consensus.cc:504] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.365796 15267 consensus_queue.cc:1035] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 20:11:08.366768 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c3dd1267b741d4a5001ac7ae1d57ca" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.367014 13137 raft_consensus.cc:3032] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:11:08.367117 13137 raft_consensus.cc:729] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.367413 13137 consensus_queue.cc:260] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.367718 13137 raft_consensus.cc:3037] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.370522 13137 raft_consensus.cc:2396] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.372176 12874 leader_election.cc:288] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 6 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:08.371956 12604 catalog_manager.cc:5484] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 5 to 6, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 6 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:08.373427 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c3dd1267b741d4a5001ac7ae1d57ca" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.373701 12746 raft_consensus.cc:3037] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.377439 12746 raft_consensus.cc:2454] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 6.
I20240501 20:11:08.379367 12812 leader_election.cc:302] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters: bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:08.380529 15013 raft_consensus.cc:2781] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:11:08.380694 15013 raft_consensus.cc:686] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 6 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.380956 15013 consensus_queue.cc:237] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.382684 12875 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1d207517fea64fe1a83a2409157ae2bd"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:08.382894 12875 raft_consensus.cc:482] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.382992 12875 raft_consensus.cc:3037] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.385952 12875 raft_consensus.cc:504] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.387216 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d207517fea64fe1a83a2409157ae2bd" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.387401 13137 raft_consensus.cc:3032] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:11:08.387480 13137 raft_consensus.cc:729] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.387707 13137 consensus_queue.cc:260] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.388015 13137 raft_consensus.cc:3037] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.391821 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d207517fea64fe1a83a2409157ae2bd" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.392118 13002 raft_consensus.cc:3037] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.392184 13137 raft_consensus.cc:2396] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:08.392704 12875 leader_election.cc:288] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 5 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:08.393882 12874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b6ba30e296834fb28bf4a88141c24826"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:08.394105 12874 raft_consensus.cc:482] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.394222 12874 raft_consensus.cc:3037] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.396926 12874 raft_consensus.cc:504] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.397998 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ba30e296834fb28bf4a88141c24826" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.398098 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ba30e296834fb28bf4a88141c24826" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.398231 12746 raft_consensus.cc:3037] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.398284 13006 raft_consensus.cc:3032] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:08.398383 13006 raft_consensus.cc:729] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:08.398680 13006 consensus_queue.cc:260] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.399076 13006 raft_consensus.cc:3037] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.402455 12746 raft_consensus.cc:2454] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 4.
I20240501 20:11:08.402599 12874 leader_election.cc:288] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 4 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:08.403774 12812 leader_election.cc:302] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters:
I20240501 20:11:08.404428 15165 raft_consensus.cc:2781] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:11:08.404867 15165 raft_consensus.cc:686] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 4 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.405325 12875 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "17276a219da74d95919c8f427c9920c5"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:08.405774 12875 raft_consensus.cc:482] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.405490 15165 consensus_queue.cc:237] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.406039 12875 raft_consensus.cc:3037] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.407094 13006 raft_consensus.cc:2454] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 4.
I20240501 20:11:08.408404 13002 raft_consensus.cc:2396] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:08.408780 12875 raft_consensus.cc:504] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.408914 12811 leader_election.cc:302] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c; no voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:08.409349 15165 raft_consensus.cc:2726] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240501 20:11:08.409884 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17276a219da74d95919c8f427c9920c5" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.410408 13002 raft_consensus.cc:3037] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.410833 12875 leader_election.cc:288] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 5 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:08.411221 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17276a219da74d95919c8f427c9920c5" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.411520 12746 raft_consensus.cc:3037] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.411916 12875 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "248dfbfcb6214244a410083e8de22743"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:08.412101 12875 raft_consensus.cc:482] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.412238 12875 raft_consensus.cc:3037] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.414932 12746 raft_consensus.cc:2396] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:08.416736 13002 raft_consensus.cc:2454] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 5.
I20240501 20:11:08.417117 12811 leader_election.cc:302] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters: 4e9d799e45f8407590cb3096f5625c62
I20240501 20:11:08.417562 15165 raft_consensus.cc:2781] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:11:08.417712 12875 raft_consensus.cc:504] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.417863 15165 raft_consensus.cc:686] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 5 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.418299 12875 leader_election.cc:288] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 5 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:08.419150 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "248dfbfcb6214244a410083e8de22743" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.419248 12875 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "47b8c200bb05478db26c0d10b25a514c"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:08.419366 13137 raft_consensus.cc:3032] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:11:08.419401 12875 raft_consensus.cc:482] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.419468 13137 raft_consensus.cc:729] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.419509 12875 raft_consensus.cc:3037] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.419751 13137 consensus_queue.cc:260] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.420060 13137 raft_consensus.cc:3037] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.422032 12875 raft_consensus.cc:504] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.422646 12875 leader_election.cc:288] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 5 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:08.423300 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b8c200bb05478db26c0d10b25a514c" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.423496 12745 raft_consensus.cc:3037] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.423676 13137 raft_consensus.cc:2396] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240501 20:11:08.424134 15165 consensus_queue.cc:237] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.425360 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b8c200bb05478db26c0d10b25a514c" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.425534 13002 raft_consensus.cc:3037] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.426137 12745 raft_consensus.cc:2396] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:08.427166 12875 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a04bef0a392e42ebae58a89d46e2f373"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:08.427412 12875 raft_consensus.cc:482] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.427565 12875 raft_consensus.cc:3037] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.431366 12875 raft_consensus.cc:504] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.431319 12604 catalog_manager.cc:5484] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 5 to 6, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 6 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.431938 12875 leader_election.cc:288] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 6 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:08.432418 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "248dfbfcb6214244a410083e8de22743" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.432603 12746 raft_consensus.cc:3037] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.433178 13002 raft_consensus.cc:2454] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 5.
I20240501 20:11:08.433414 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04bef0a392e42ebae58a89d46e2f373" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.433595 12745 raft_consensus.cc:3037] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.434072 12811 leader_election.cc:302] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters: 4e9d799e45f8407590cb3096f5625c62
I20240501 20:11:08.434520 15013 raft_consensus.cc:2781] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:11:08.434751 15013 raft_consensus.cc:686] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 5 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.435021 15013 consensus_queue.cc:237] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.436230 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04bef0a392e42ebae58a89d46e2f373" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.436446 13137 raft_consensus.cc:3032] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:11:08.436545 13137 raft_consensus.cc:729] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.436544 12745 raft_consensus.cc:2454] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 6.
I20240501 20:11:08.436841 13137 consensus_queue.cc:260] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.437146 13137 raft_consensus.cc:3037] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.437759 12812 leader_election.cc:302] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters:
I20240501 20:11:08.438030 14437 raft_consensus.cc:2781] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:11:08.438472 12875 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9a992a1cf4484b298200677ee8b56424"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:08.438704 12875 raft_consensus.cc:482] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.438861 12875 raft_consensus.cc:3037] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.439227 12746 raft_consensus.cc:2454] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 5.
I20240501 20:11:08.439680 12812 leader_election.cc:302] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters: bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:08.452973 12875 raft_consensus.cc:504] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.453819 12875 leader_election.cc:288] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 6 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:08.454972 12874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b96f18923f9d41878cb01202e77be033"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:08.455185 12874 raft_consensus.cc:482] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.455327 12874 raft_consensus.cc:3037] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.456228 13137 raft_consensus.cc:2396] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.459233 12874 raft_consensus.cc:504] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.460104 12874 leader_election.cc:288] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 5 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:08.460623 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96f18923f9d41878cb01202e77be033" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.460850 13002 raft_consensus.cc:3037] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.462947 12874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4488bf7917574118915a9355219528ee"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:08.463153 12874 raft_consensus.cc:482] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.463276 12874 raft_consensus.cc:3037] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.464529 13002 raft_consensus.cc:2396] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:08.467196 12874 raft_consensus.cc:504] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.468015 12874 leader_election.cc:288] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 4 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:08.468401 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4488bf7917574118915a9355219528ee" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.468786 13002 raft_consensus.cc:3032] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:08.468907 13002 raft_consensus.cc:729] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:08.469056 12874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d85b0f46583649e9bee3f433ba6f928e"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:08.469267 12874 raft_consensus.cc:482] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.469182 13002 consensus_queue.cc:260] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.469405 12874 raft_consensus.cc:3037] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.469544 13002 raft_consensus.cc:3037] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.470772 15314 raft_consensus.cc:2781] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:11:08.470966 15314 raft_consensus.cc:686] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 5 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.471369 15314 consensus_queue.cc:237] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.472316 13002 raft_consensus.cc:2454] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 4.
I20240501 20:11:08.472796 12811 leader_election.cc:302] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:08.473125 15314 raft_consensus.cc:2781] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:11:08.473323 15314 raft_consensus.cc:686] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 4 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.473708 14437 raft_consensus.cc:686] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 6 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.474323 14437 consensus_queue.cc:237] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.473631 12874 raft_consensus.cc:504] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.476174 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96f18923f9d41878cb01202e77be033" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.476374 13134 raft_consensus.cc:3037] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.476214 12874 leader_election.cc:288] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 4 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:08.475593 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a992a1cf4484b298200677ee8b56424" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.477067 13137 raft_consensus.cc:3032] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:11:08.477030 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85b0f46583649e9bee3f433ba6f928e" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.477173 13137 raft_consensus.cc:729] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.477236 13002 raft_consensus.cc:3032] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 20:11:08.477327 13002 raft_consensus.cc:729] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:08.477452 13137 consensus_queue.cc:260] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.477571 13002 consensus_queue.cc:260] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.477788 13137 raft_consensus.cc:3037] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.477952 13002 raft_consensus.cc:3037] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.478874 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a992a1cf4484b298200677ee8b56424" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.479099 12746 raft_consensus.cc:3037] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.480682 13137 raft_consensus.cc:2396] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.480650 13002 raft_consensus.cc:2454] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 4.
I20240501 20:11:08.481125 12811 leader_election.cc:302] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:08.481496 15315 raft_consensus.cc:2781] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Leader election won for term 4
I20240501 20:11:08.481638 15315 raft_consensus.cc:686] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 4 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.481938 12746 raft_consensus.cc:2454] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 6.
I20240501 20:11:08.482003 15315 consensus_queue.cc:237] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.483773 12812 leader_election.cc:302] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c; no voters: bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:08.483796 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4488bf7917574118915a9355219528ee" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.484014 12745 raft_consensus.cc:3037] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.484302 15165 raft_consensus.cc:2781] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:11:08.484423 15165 raft_consensus.cc:686] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 6 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.484727 15165 consensus_queue.cc:237] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.487964 13134 raft_consensus.cc:2454] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 5.
I20240501 20:11:08.488711 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85b0f46583649e9bee3f433ba6f928e" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.489059 12746 raft_consensus.cc:3037] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [term 3 FOLLOWER]: Advancing to term 4
I20240501 20:11:08.493386 12746 raft_consensus.cc:2454] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 4.
I20240501 20:11:08.493741 12812 leader_election.cc:302] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231; no voters: b4720929b44c4560bfdf08107e137362
I20240501 20:11:08.494590 12745 raft_consensus.cc:2454] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 4.
I20240501 20:11:08.494781 14437 raft_consensus.cc:2781] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:11:08.494940 12874 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e89e2dfe0abc487593a61f9f1b58ac65"
dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
from {username='slave'} at 127.0.0.1:51834
I20240501 20:11:08.495111 14437 raft_consensus.cc:686] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 5 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.495164 12874 raft_consensus.cc:482] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.495280 12874 raft_consensus.cc:3037] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.495460 14437 consensus_queue.cc:237] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.498175 12874 raft_consensus.cc:504] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.473632 15314 consensus_queue.cc:237] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.498869 12874 leader_election.cc:288] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 5 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629)
I20240501 20:11:08.499409 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89e2dfe0abc487593a61f9f1b58ac65" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.499493 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89e2dfe0abc487593a61f9f1b58ac65" candidate_uuid: "77bf705599644326b6e71b5caf32fc7c" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.499624 13002 raft_consensus.cc:3037] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.499712 12746 raft_consensus.cc:3037] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.502516 13002 raft_consensus.cc:2454] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77bf705599644326b6e71b5caf32fc7c in term 5.
I20240501 20:11:08.502956 12811 leader_election.cc:302] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:08.503238 15314 raft_consensus.cc:2781] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:11:08.503365 15314 raft_consensus.cc:686] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 5 LEADER]: Becoming Leader. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.503594 15314 consensus_queue.cc:237] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.504832 12604 catalog_manager.cc:5484] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 5 to 6, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 6 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:08.505237 12604 catalog_manager.cc:5484] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 3 to 4, leader changed from b4720929b44c4560bfdf08107e137362 (127.29.219.67) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 4 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:08.505538 12604 catalog_manager.cc:5484] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 3 to 5, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 5 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:08.505800 12604 catalog_manager.cc:5484] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 4 to 5, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 5 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.506096 12604 catalog_manager.cc:5484] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 3 to 5, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 5 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:08.506379 12604 catalog_manager.cc:5484] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 3 to 4, leader changed from b4720929b44c4560bfdf08107e137362 (127.29.219.67) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 4 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.502609 12746 raft_consensus.cc:2396] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 77bf705599644326b6e71b5caf32fc7c for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:08.514789 12604 catalog_manager.cc:5484] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 3 to 5, leader changed from b4720929b44c4560bfdf08107e137362 (127.29.219.67) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 5 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.515430 12604 catalog_manager.cc:5484] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 3 to 4, leader changed from b4720929b44c4560bfdf08107e137362 (127.29.219.67) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 4 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:08.515985 12604 catalog_manager.cc:5484] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 3 to 5, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 5 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.516443 12604 catalog_manager.cc:5484] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c reported cstate change: term changed from 5 to 6, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 77bf705599644326b6e71b5caf32fc7c (127.29.219.66). New cstate: current_term: 6 leader_uuid: "77bf705599644326b6e71b5caf32fc7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:08.519539 12746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9a992a1cf4484b298200677ee8b56424"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:08.519758 12746 raft_consensus.cc:482] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.519883 12746 raft_consensus.cc:3037] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.521889 12746 raft_consensus.cc:504] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.522514 12746 leader_election.cc:288] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 7 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:08.523074 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a992a1cf4484b298200677ee8b56424" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.523285 12874 raft_consensus.cc:3032] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 6 LEADER]: Stepping down as leader of term 6
I20240501 20:11:08.523381 12874 raft_consensus.cc:729] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.523638 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a992a1cf4484b298200677ee8b56424" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.523625 12874 consensus_queue.cc:260] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.523830 13134 raft_consensus.cc:3037] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.523906 12874 raft_consensus.cc:3037] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.526543 13134 raft_consensus.cc:2396] T 9a992a1cf4484b298200677ee8b56424 P bb1be87a7aee48a4b405b9d209c61231 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:08.526700 12874 raft_consensus.cc:2396] T 9a992a1cf4484b298200677ee8b56424 P 77bf705599644326b6e71b5caf32fc7c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:08.528695 12681 leader_election.cc:302] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62; no voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:08.529176 15239 raft_consensus.cc:2726] T 9a992a1cf4484b298200677ee8b56424 P 4e9d799e45f8407590cb3096f5625c62 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 20:11:08.530331 12746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3eac603886e4a84877d71ccebd0dff0"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:08.530553 12746 raft_consensus.cc:482] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.530685 12746 raft_consensus.cc:3037] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.532680 12746 raft_consensus.cc:504] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.533285 12746 leader_election.cc:288] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 7 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:08.533829 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3eac603886e4a84877d71ccebd0dff0" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.533977 12874 raft_consensus.cc:3032] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 6 LEADER]: Stepping down as leader of term 6
I20240501 20:11:08.534039 12874 raft_consensus.cc:729] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.534281 12874 consensus_queue.cc:260] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.534512 12874 raft_consensus.cc:3037] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.535104 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3eac603886e4a84877d71ccebd0dff0" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.535321 13134 raft_consensus.cc:3037] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.537635 12874 raft_consensus.cc:2396] T d3eac603886e4a84877d71ccebd0dff0 P 77bf705599644326b6e71b5caf32fc7c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:08.537995 13134 raft_consensus.cc:2396] T d3eac603886e4a84877d71ccebd0dff0 P bb1be87a7aee48a4b405b9d209c61231 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:08.538478 12681 leader_election.cc:302] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62; no voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:08.538725 12746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f22d4331ed6f4e86b85a9919c219ab84"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:08.538933 15239 raft_consensus.cc:2726] T d3eac603886e4a84877d71ccebd0dff0 P 4e9d799e45f8407590cb3096f5625c62 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 20:11:08.538929 12746 raft_consensus.cc:482] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.539099 12746 raft_consensus.cc:3037] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.541577 12746 raft_consensus.cc:504] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.542279 12746 leader_election.cc:288] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 6 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:08.542783 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22d4331ed6f4e86b85a9919c219ab84" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.543037 13002 raft_consensus.cc:3037] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.543239 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22d4331ed6f4e86b85a9919c219ab84" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.543360 12745 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ddb529dd5534f8bb66562e8467c1f12"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:08.543465 13134 raft_consensus.cc:3037] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.543560 12745 raft_consensus.cc:482] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.543653 12745 raft_consensus.cc:3037] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.545900 13002 raft_consensus.cc:2396] T f22d4331ed6f4e86b85a9919c219ab84 P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240501 20:11:08.546284 13134 raft_consensus.cc:2454] T f22d4331ed6f4e86b85a9919c219ab84 P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 6.
I20240501 20:11:08.546319 12745 raft_consensus.cc:504] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.546801 12681 leader_election.cc:302] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:08.547076 12745 leader_election.cc:288] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 5 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:08.547111 15239 raft_consensus.cc:2781] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:11:08.547760 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ddb529dd5534f8bb66562e8467c1f12" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.547981 13002 raft_consensus.cc:3037] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.548645 12745 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "436330ea0516443799906257122b38af"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:08.548902 12745 raft_consensus.cc:482] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.549158 12745 raft_consensus.cc:3037] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.550186 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ddb529dd5534f8bb66562e8467c1f12" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.550457 12874 raft_consensus.cc:3032] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:11:08.550590 12874 raft_consensus.cc:729] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.551208 12874 consensus_queue.cc:260] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.551631 12874 raft_consensus.cc:3037] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.552501 12745 raft_consensus.cc:504] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.553262 12745 leader_election.cc:288] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 6 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:08.553756 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436330ea0516443799906257122b38af" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.553975 13006 raft_consensus.cc:3037] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.554117 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436330ea0516443799906257122b38af" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.554322 13134 raft_consensus.cc:3037] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.556366 12874 raft_consensus.cc:2396] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.556851 13006 raft_consensus.cc:2396] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240501 20:11:08.556996 13002 raft_consensus.cc:2454] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 5.
I20240501 20:11:08.557721 12680 leader_election.cc:302] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters: 77bf705599644326b6e71b5caf32fc7c
I20240501 20:11:08.557721 12745 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "66c3dd1267b741d4a5001ac7ae1d57ca"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:08.558044 12745 raft_consensus.cc:482] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.558171 12745 raft_consensus.cc:3037] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.560629 13134 raft_consensus.cc:2454] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 6.
I20240501 20:11:08.560928 15326 raft_consensus.cc:2781] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:11:08.561095 12745 raft_consensus.cc:504] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.561180 12681 leader_election.cc:302] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, bb1be87a7aee48a4b405b9d209c61231; no voters: b4720929b44c4560bfdf08107e137362
I20240501 20:11:08.561179 15326 raft_consensus.cc:686] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 5 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.561715 15326 consensus_queue.cc:237] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.562743 15326 raft_consensus.cc:2781] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:11:08.561902 12745 leader_election.cc:288] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 7 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:08.567155 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c3dd1267b741d4a5001ac7ae1d57ca" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.567416 13137 raft_consensus.cc:3037] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.567409 15326 raft_consensus.cc:686] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 6 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.568003 12745 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "50561f1e0c5b47a78abb1fba84e725fa"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:08.568204 12745 raft_consensus.cc:482] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.568324 12745 raft_consensus.cc:3037] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.569804 13137 raft_consensus.cc:2396] T 66c3dd1267b741d4a5001ac7ae1d57ca P bb1be87a7aee48a4b405b9d209c61231 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:08.563952 15239 raft_consensus.cc:686] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [term 6 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.570660 15239 consensus_queue.cc:237] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.570747 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66c3dd1267b741d4a5001ac7ae1d57ca" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.570966 12874 raft_consensus.cc:3032] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 6 LEADER]: Stepping down as leader of term 6
I20240501 20:11:08.571090 12874 raft_consensus.cc:729] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.571092 12745 raft_consensus.cc:504] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.571406 12874 consensus_queue.cc:260] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.571738 12874 raft_consensus.cc:3037] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.572121 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50561f1e0c5b47a78abb1fba84e725fa" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.572403 13002 raft_consensus.cc:3037] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.575482 12874 raft_consensus.cc:2396] T 66c3dd1267b741d4a5001ac7ae1d57ca P 77bf705599644326b6e71b5caf32fc7c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:08.575784 13002 raft_consensus.cc:2454] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 6.
I20240501 20:11:08.575968 12681 leader_election.cc:302] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62; no voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:08.576001 15326 consensus_queue.cc:237] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.576704 12680 leader_election.cc:302] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:08.577060 15328 raft_consensus.cc:2781] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:11:08.576267 15327 raft_consensus.cc:2726] T 66c3dd1267b741d4a5001ac7ae1d57ca P 4e9d799e45f8407590cb3096f5625c62 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 20:11:08.577451 12745 leader_election.cc:288] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 6 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:08.578464 12745 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5f728b714ca142648ba5aa3459f7ae80"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:08.578733 12745 raft_consensus.cc:482] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.578886 12745 raft_consensus.cc:3037] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.579083 12604 catalog_manager.cc:5484] T f22d4331ed6f4e86b85a9919c219ab84 P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 4 to 6, leader changed from b4720929b44c4560bfdf08107e137362 (127.29.219.67) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 6 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.580679 15328 raft_consensus.cc:686] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 6 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.580967 15328 consensus_queue.cc:237] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.581959 12745 raft_consensus.cc:504] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.582634 12745 leader_election.cc:288] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 7 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:08.583644 12745 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c1211cfd40674438a14a49818863b3f0"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:08.583822 12745 raft_consensus.cc:482] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.583948 12745 raft_consensus.cc:3037] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.586660 12745 raft_consensus.cc:504] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.587917 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1211cfd40674438a14a49818863b3f0" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.588196 13002 raft_consensus.cc:3037] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.591745 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f728b714ca142648ba5aa3459f7ae80" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.591967 12874 raft_consensus.cc:3032] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 6 LEADER]: Stepping down as leader of term 6
I20240501 20:11:08.591998 13002 raft_consensus.cc:2454] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 6.
I20240501 20:11:08.592065 12874 raft_consensus.cc:729] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.592320 12874 consensus_queue.cc:260] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.592455 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f728b714ca142648ba5aa3459f7ae80" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.592648 12874 raft_consensus.cc:3037] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.592670 13137 raft_consensus.cc:3037] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.592821 12680 leader_election.cc:302] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:08.593313 15239 raft_consensus.cc:2781] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:11:08.593468 15239 raft_consensus.cc:686] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 6 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.594291 15239 consensus_queue.cc:237] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.595556 12874 raft_consensus.cc:2396] T 5f728b714ca142648ba5aa3459f7ae80 P 77bf705599644326b6e71b5caf32fc7c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:08.595695 13137 raft_consensus.cc:2396] T 5f728b714ca142648ba5aa3459f7ae80 P bb1be87a7aee48a4b405b9d209c61231 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:08.595773 12745 leader_election.cc:288] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 6 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:08.596079 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50561f1e0c5b47a78abb1fba84e725fa" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.596263 13134 raft_consensus.cc:3032] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:11:08.596371 13134 raft_consensus.cc:729] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.596601 13134 consensus_queue.cc:260] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.596909 13134 raft_consensus.cc:3037] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.597031 12745 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e00b9b3645a04a4686e5759bd8201eca"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:08.597311 12745 raft_consensus.cc:482] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.597435 12745 raft_consensus.cc:3037] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.597848 12604 catalog_manager.cc:5484] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 4 to 5, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 5 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.600029 13134 raft_consensus.cc:2396] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.598470 12604 catalog_manager.cc:5484] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 5 to 6, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 6 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.600822 12604 catalog_manager.cc:5484] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 4 to 6, leader changed from b4720929b44c4560bfdf08107e137362 (127.29.219.67) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 6 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.601954 12745 raft_consensus.cc:504] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.602571 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1211cfd40674438a14a49818863b3f0" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.602839 12874 raft_consensus.cc:3032] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:11:08.602957 12874 raft_consensus.cc:729] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.603121 12745 leader_election.cc:288] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 7 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:08.603257 12874 consensus_queue.cc:260] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.603597 12874 raft_consensus.cc:3037] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.604485 12745 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b6ba30e296834fb28bf4a88141c24826"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:08.604786 12745 raft_consensus.cc:482] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.605000 12745 raft_consensus.cc:3037] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.606496 12746 raft_consensus.cc:1261] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Refusing update from remote peer bb1be87a7aee48a4b405b9d209c61231: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240501 20:11:08.607842 12874 raft_consensus.cc:2396] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.608461 15140 consensus_queue.cc:1035] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:08.608738 12681 leader_election.cc:302] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62; no voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:08.608891 12745 raft_consensus.cc:504] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.610054 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ba30e296834fb28bf4a88141c24826" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.610198 15328 raft_consensus.cc:2726] T 5f728b714ca142648ba5aa3459f7ae80 P 4e9d799e45f8407590cb3096f5625c62 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 20:11:08.610286 13002 raft_consensus.cc:3037] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.611366 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00b9b3645a04a4686e5759bd8201eca" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.611565 13134 raft_consensus.cc:3037] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.613415 13002 raft_consensus.cc:2454] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 5.
I20240501 20:11:08.613888 12680 leader_election.cc:302] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:08.614171 15328 raft_consensus.cc:2781] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:11:08.614317 15328 raft_consensus.cc:686] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 5 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.614607 15328 consensus_queue.cc:237] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.614863 13134 raft_consensus.cc:2396] T e00b9b3645a04a4686e5759bd8201eca P bb1be87a7aee48a4b405b9d209c61231 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:08.615670 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e00b9b3645a04a4686e5759bd8201eca" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.615865 12874 raft_consensus.cc:3032] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 6 LEADER]: Stepping down as leader of term 6
I20240501 20:11:08.616060 12874 raft_consensus.cc:729] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.616443 12874 consensus_queue.cc:260] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.616822 12874 raft_consensus.cc:3037] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.620626 12874 raft_consensus.cc:2396] T e00b9b3645a04a4686e5759bd8201eca P 77bf705599644326b6e71b5caf32fc7c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:08.621695 12875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ba30e296834fb28bf4a88141c24826" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.621961 12875 raft_consensus.cc:3032] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:11:08.622067 12875 raft_consensus.cc:729] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.622393 12875 consensus_queue.cc:260] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.622766 12875 raft_consensus.cc:3037] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.622723 12681 leader_election.cc:302] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62; no voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:08.623307 15328 raft_consensus.cc:2726] T e00b9b3645a04a4686e5759bd8201eca P 4e9d799e45f8407590cb3096f5625c62 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 20:11:08.625016 12745 leader_election.cc:288] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 5 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:08.626374 12875 raft_consensus.cc:2396] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.626530 12746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dafd65d56bb243c78cb4490d7da7b18a"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:08.626734 12746 raft_consensus.cc:482] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.626856 12746 raft_consensus.cc:3037] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.629611 12746 raft_consensus.cc:504] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.630704 13006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dafd65d56bb243c78cb4490d7da7b18a" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.630882 13002 raft_consensus.cc:1261] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Refusing update from remote peer bb1be87a7aee48a4b405b9d209c61231: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240501 20:11:08.630946 13006 raft_consensus.cc:3037] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.631498 15152 consensus_queue.cc:1035] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 20:11:08.631551 12604 catalog_manager.cc:5484] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 5 to 6, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 6 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.632520 12604 catalog_manager.cc:5484] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 4 to 5, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 5 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.633725 13006 raft_consensus.cc:2396] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240501 20:11:08.633798 12746 leader_election.cc:288] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 6 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:08.634858 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dafd65d56bb243c78cb4490d7da7b18a" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.635099 13134 raft_consensus.cc:3037] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.637926 13134 raft_consensus.cc:2454] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 6.
I20240501 20:11:08.637931 12746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "17276a219da74d95919c8f427c9920c5"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:08.638185 12746 raft_consensus.cc:482] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.638312 12746 raft_consensus.cc:3037] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.638397 12681 leader_election.cc:302] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, bb1be87a7aee48a4b405b9d209c61231; no voters: b4720929b44c4560bfdf08107e137362
I20240501 20:11:08.638844 15239 raft_consensus.cc:2781] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:11:08.639094 15239 raft_consensus.cc:686] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 6 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.639389 15239 consensus_queue.cc:237] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.641573 12746 raft_consensus.cc:504] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.642315 12746 leader_election.cc:288] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 6 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:08.642859 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17276a219da74d95919c8f427c9920c5" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.643095 12874 raft_consensus.cc:3032] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:11:08.643102 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17276a219da74d95919c8f427c9920c5" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.643214 12874 raft_consensus.cc:729] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.643473 12746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "248dfbfcb6214244a410083e8de22743"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:08.643486 12874 consensus_queue.cc:260] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.643631 12746 raft_consensus.cc:482] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.643738 12746 raft_consensus.cc:3037] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.643811 12874 raft_consensus.cc:3037] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.644922 13002 raft_consensus.cc:3037] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.646724 12874 raft_consensus.cc:2396] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.644690 12601 catalog_manager.cc:5484] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 4 to 6, leader changed from b4720929b44c4560bfdf08107e137362 (127.29.219.67) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 6 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.646592 12746 raft_consensus.cc:504] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.647877 12746 leader_election.cc:288] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 6 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:08.648849 12746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "47b8c200bb05478db26c0d10b25a514c"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:08.648803 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "248dfbfcb6214244a410083e8de22743" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.649026 13134 raft_consensus.cc:3037] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.649324 12746 raft_consensus.cc:482] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.649483 12746 raft_consensus.cc:3037] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.650121 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "248dfbfcb6214244a410083e8de22743" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.650305 12874 raft_consensus.cc:3032] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:11:08.650413 12874 raft_consensus.cc:729] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.650725 12874 consensus_queue.cc:260] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.652581 12746 raft_consensus.cc:504] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.653017 13134 raft_consensus.cc:2396] T 248dfbfcb6214244a410083e8de22743 P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240501 20:11:08.653328 12874 raft_consensus.cc:3037] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.653811 13004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b8c200bb05478db26c0d10b25a514c" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.654035 13004 raft_consensus.cc:3037] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.654716 12875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b8c200bb05478db26c0d10b25a514c" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.654949 12875 raft_consensus.cc:3032] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:11:08.655028 12875 raft_consensus.cc:729] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.655303 12875 consensus_queue.cc:260] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.655560 12875 raft_consensus.cc:3037] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.656154 12874 raft_consensus.cc:2396] T 248dfbfcb6214244a410083e8de22743 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240501 20:11:08.657653 12875 raft_consensus.cc:2396] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.657783 13004 raft_consensus.cc:2454] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 6.
I20240501 20:11:08.653961 13002 raft_consensus.cc:2454] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 6.
I20240501 20:11:08.658303 12680 leader_election.cc:302] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:08.658650 15239 raft_consensus.cc:2781] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:11:08.658685 12680 leader_election.cc:302] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters: 77bf705599644326b6e71b5caf32fc7c
I20240501 20:11:08.658816 15239 raft_consensus.cc:686] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 6 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.658983 15328 raft_consensus.cc:2781] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:11:08.659157 15328 raft_consensus.cc:686] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 6 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.659098 15239 consensus_queue.cc:237] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.659425 15328 consensus_queue.cc:237] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.659471 12681 leader_election.cc:302] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62; no voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:08.659829 15326 raft_consensus.cc:2726] T 248dfbfcb6214244a410083e8de22743 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 20:11:08.660360 12746 leader_election.cc:288] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 6 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:08.661484 12746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a04bef0a392e42ebae58a89d46e2f373"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:08.661687 12746 raft_consensus.cc:482] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.661824 12746 raft_consensus.cc:3037] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.663360 12601 catalog_manager.cc:5484] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 5 to 6, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 6 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.663882 12746 raft_consensus.cc:504] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.664876 12875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04bef0a392e42ebae58a89d46e2f373" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.665088 12875 raft_consensus.cc:3032] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 6 LEADER]: Stepping down as leader of term 6
I20240501 20:11:08.665175 12875 raft_consensus.cc:729] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.665434 12875 consensus_queue.cc:260] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.665628 12875 raft_consensus.cc:3037] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.667659 12875 raft_consensus.cc:2396] T a04bef0a392e42ebae58a89d46e2f373 P 77bf705599644326b6e71b5caf32fc7c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:08.668043 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04bef0a392e42ebae58a89d46e2f373" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.668330 13134 raft_consensus.cc:3037] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.670886 13134 raft_consensus.cc:2396] T a04bef0a392e42ebae58a89d46e2f373 P bb1be87a7aee48a4b405b9d209c61231 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:08.671108 12746 leader_election.cc:288] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 7 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:08.672122 12746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4488bf7917574118915a9355219528ee"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:08.672358 12746 raft_consensus.cc:482] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.672472 12746 raft_consensus.cc:3037] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.672899 12681 leader_election.cc:302] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62; no voters: 77bf705599644326b6e71b5caf32fc7c, bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:08.673245 15328 raft_consensus.cc:2726] T a04bef0a392e42ebae58a89d46e2f373 P 4e9d799e45f8407590cb3096f5625c62 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 20:11:08.674657 12601 catalog_manager.cc:5484] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 5 to 6, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 6 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.675357 12746 raft_consensus.cc:504] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.676024 12746 leader_election.cc:288] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 5 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:08.676476 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4488bf7917574118915a9355219528ee" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.676674 13002 raft_consensus.cc:3037] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.677187 12746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "70cd215b4f9445499e14c9fd18f7f7fe"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:08.677384 12746 raft_consensus.cc:482] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.677511 12746 raft_consensus.cc:3037] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.678717 13002 raft_consensus.cc:2454] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 5.
I20240501 20:11:08.679194 12680 leader_election.cc:302] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:08.679500 15328 raft_consensus.cc:2781] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:11:08.679677 15328 raft_consensus.cc:686] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 5 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.680030 15328 consensus_queue.cc:237] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.680130 12746 raft_consensus.cc:504] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.680689 12746 leader_election.cc:288] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 6 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:08.681574 12875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4488bf7917574118915a9355219528ee" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.681797 12875 raft_consensus.cc:3032] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:11:08.681859 12875 raft_consensus.cc:729] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.682119 12875 consensus_queue.cc:260] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.682330 12875 raft_consensus.cc:3037] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.683300 12746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d85b0f46583649e9bee3f433ba6f928e"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:08.683502 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70cd215b4f9445499e14c9fd18f7f7fe" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.683717 13002 raft_consensus.cc:3037] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.683769 12746 raft_consensus.cc:482] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.684005 12746 raft_consensus.cc:3037] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.684760 12875 raft_consensus.cc:2396] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.685204 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70cd215b4f9445499e14c9fd18f7f7fe" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.685413 13134 raft_consensus.cc:3037] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.687836 13002 raft_consensus.cc:2396] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240501 20:11:08.687999 12746 raft_consensus.cc:504] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.689028 13134 raft_consensus.cc:2454] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 6.
I20240501 20:11:08.689041 12601 catalog_manager.cc:5484] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 4 to 5, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 5 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.689386 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85b0f46583649e9bee3f433ba6f928e" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.689568 13002 raft_consensus.cc:3037] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.689627 12681 leader_election.cc:302] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, bb1be87a7aee48a4b405b9d209c61231; no voters: b4720929b44c4560bfdf08107e137362
I20240501 20:11:08.690208 15328 raft_consensus.cc:2781] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:11:08.690506 15328 raft_consensus.cc:686] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 6 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.690917 15328 consensus_queue.cc:237] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.692179 13002 raft_consensus.cc:2454] T d85b0f46583649e9bee3f433ba6f928e P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 5.
I20240501 20:11:08.692591 12680 leader_election.cc:302] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:08.693120 15328 raft_consensus.cc:2781] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:11:08.693289 15328 raft_consensus.cc:686] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [term 5 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.693532 15328 consensus_queue.cc:237] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.694074 12746 leader_election.cc:288] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 5 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:08.694197 12875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d85b0f46583649e9bee3f433ba6f928e" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.694435 12875 raft_consensus.cc:3032] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:11:08.694537 12875 raft_consensus.cc:729] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.694877 12875 consensus_queue.cc:260] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.695343 12875 raft_consensus.cc:3037] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.695906 12746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e89e2dfe0abc487593a61f9f1b58ac65"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:08.696094 12746 raft_consensus.cc:482] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.696213 12746 raft_consensus.cc:3037] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.698276 12875 raft_consensus.cc:2396] T d85b0f46583649e9bee3f433ba6f928e P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.698868 12746 raft_consensus.cc:504] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.700613 12875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89e2dfe0abc487593a61f9f1b58ac65" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.700834 12875 raft_consensus.cc:3032] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:11:08.700928 12875 raft_consensus.cc:729] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.701298 12875 consensus_queue.cc:260] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.701174 12601 catalog_manager.cc:5484] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 4 to 6, leader changed from b4720929b44c4560bfdf08107e137362 (127.29.219.67) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 6 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.701586 12875 raft_consensus.cc:3037] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.701788 12601 catalog_manager.cc:5484] T d85b0f46583649e9bee3f433ba6f928e P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 4 to 5, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 5 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.703089 12746 leader_election.cc:288] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 6 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:08.703819 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89e2dfe0abc487593a61f9f1b58ac65" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.704083 13002 raft_consensus.cc:3037] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.705585 12875 raft_consensus.cc:2396] T e89e2dfe0abc487593a61f9f1b58ac65 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.708659 13002 raft_consensus.cc:2454] T e89e2dfe0abc487593a61f9f1b58ac65 P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 6.
I20240501 20:11:08.709249 12680 leader_election.cc:302] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters: 77bf705599644326b6e71b5caf32fc7c
I20240501 20:11:08.710592 12746 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5af8328d0e9248acb93e3707ffde7f8a"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:08.710788 15328 raft_consensus.cc:2781] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:11:08.711091 15328 raft_consensus.cc:686] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [term 6 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.711519 15328 consensus_queue.cc:237] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.712854 12746 raft_consensus.cc:482] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.713053 12746 raft_consensus.cc:3037] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.715673 12601 catalog_manager.cc:5484] T e89e2dfe0abc487593a61f9f1b58ac65 P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 5 to 6, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 6 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:08.717583 12746 raft_consensus.cc:504] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.718703 12746 leader_election.cc:288] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 5 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701)
I20240501 20:11:08.719156 13134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af8328d0e9248acb93e3707ffde7f8a" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.719270 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af8328d0e9248acb93e3707ffde7f8a" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.719417 13134 raft_consensus.cc:3037] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.719478 13002 raft_consensus.cc:3037] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.719921 12745 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e1dbb57f13d348cc996b5248b7623b8e"
dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
from {username='slave'} at 127.0.0.1:36538
I20240501 20:11:08.720158 12745 raft_consensus.cc:482] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.720295 12745 raft_consensus.cc:3037] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.722385 13134 raft_consensus.cc:2454] T 5af8328d0e9248acb93e3707ffde7f8a P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 5.
I20240501 20:11:08.722519 13002 raft_consensus.cc:2396] T 5af8328d0e9248acb93e3707ffde7f8a P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4e9d799e45f8407590cb3096f5625c62 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 20:11:08.723155 12681 leader_election.cc:302] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, bb1be87a7aee48a4b405b9d209c61231; no voters: b4720929b44c4560bfdf08107e137362
I20240501 20:11:08.723892 12745 raft_consensus.cc:504] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.724463 15328 raft_consensus.cc:2781] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:11:08.724622 15328 raft_consensus.cc:686] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [term 5 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.724843 12745 leader_election.cc:288] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 5 election: Requested vote from peers b4720929b44c4560bfdf08107e137362 (127.29.219.67:42701), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:08.724946 15328 consensus_queue.cc:237] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } }
I20240501 20:11:08.726497 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1dbb57f13d348cc996b5248b7623b8e" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.726778 13137 raft_consensus.cc:3032] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 20:11:08.726907 13137 raft_consensus.cc:729] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: bb1be87a7aee48a4b405b9d209c61231, State: Running, Role: LEADER
I20240501 20:11:08.727263 13137 consensus_queue.cc:260] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.727437 13002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1dbb57f13d348cc996b5248b7623b8e" candidate_uuid: "4e9d799e45f8407590cb3096f5625c62" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "b4720929b44c4560bfdf08107e137362"
I20240501 20:11:08.727663 13002 raft_consensus.cc:3037] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.727679 13137 raft_consensus.cc:3037] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 4 FOLLOWER]: Advancing to term 5
I20240501 20:11:08.730581 13002 raft_consensus.cc:2454] T e1dbb57f13d348cc996b5248b7623b8e P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 5.
I20240501 20:11:08.731114 12680 leader_election.cc:302] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e9d799e45f8407590cb3096f5625c62, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:08.731691 15328 raft_consensus.cc:2781] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 20:11:08.731874 15328 raft_consensus.cc:686] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [term 5 LEADER]: Becoming Leader. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.732129 15328 consensus_queue.cc:237] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.730581 13137 raft_consensus.cc:2454] T e1dbb57f13d348cc996b5248b7623b8e P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e9d799e45f8407590cb3096f5625c62 in term 5.
I20240501 20:11:08.733389 12601 catalog_manager.cc:5484] T 5af8328d0e9248acb93e3707ffde7f8a P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 3 to 5, leader changed from b4720929b44c4560bfdf08107e137362 (127.29.219.67) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 5 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:08.741154 12604 catalog_manager.cc:5484] T e1dbb57f13d348cc996b5248b7623b8e P 4e9d799e45f8407590cb3096f5625c62 reported cstate change: term changed from 4 to 5, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65). New cstate: current_term: 5 leader_uuid: "4e9d799e45f8407590cb3096f5625c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.741843 13002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e077004855949878d554a2053131790"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:08.742125 13002 raft_consensus.cc:482] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.742322 13002 raft_consensus.cc:3037] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.745285 13002 raft_consensus.cc:504] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.746213 13002 leader_election.cc:288] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 7 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:08.746932 12875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e077004855949878d554a2053131790" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.747226 12875 raft_consensus.cc:3037] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.750157 12875 raft_consensus.cc:2454] T 4e077004855949878d554a2053131790 P 77bf705599644326b6e71b5caf32fc7c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 7.
I20240501 20:11:08.750679 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e077004855949878d554a2053131790" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.750754 12942 leader_election.cc:302] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:08.750917 13137 raft_consensus.cc:3037] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.751581 15267 raft_consensus.cc:2781] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 20:11:08.751732 15267 raft_consensus.cc:686] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [term 7 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:08.752061 15267 consensus_queue.cc:237] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.753754 13137 raft_consensus.cc:2396] T 4e077004855949878d554a2053131790 P bb1be87a7aee48a4b405b9d209c61231 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 20:11:08.754841 13002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c1211cfd40674438a14a49818863b3f0"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:08.755100 13002 raft_consensus.cc:482] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.755227 13002 raft_consensus.cc:3037] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.758162 13002 raft_consensus.cc:504] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.758944 13002 leader_election.cc:288] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 7 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:08.761837 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1211cfd40674438a14a49818863b3f0" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.762157 12746 raft_consensus.cc:3032] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 20:11:08.762354 12746 raft_consensus.cc:729] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.762748 12746 consensus_queue.cc:260] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.763255 12746 raft_consensus.cc:3037] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.765453 12746 raft_consensus.cc:2396] T c1211cfd40674438a14a49818863b3f0 P 4e9d799e45f8407590cb3096f5625c62 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:08.766714 12601 catalog_manager.cc:5484] T 4e077004855949878d554a2053131790 P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 5 to 7, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 7 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.768692 13002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eeba0c03d83a433d9a519ba9fe29b53a"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:08.768779 12875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1211cfd40674438a14a49818863b3f0" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.768906 13002 raft_consensus.cc:482] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.768990 12875 raft_consensus.cc:3037] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.769035 13002 raft_consensus.cc:3037] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.771323 13002 raft_consensus.cc:504] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.771915 12875 raft_consensus.cc:2396] T c1211cfd40674438a14a49818863b3f0 P 77bf705599644326b6e71b5caf32fc7c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:08.772173 13002 leader_election.cc:288] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 7 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:08.772403 12942 leader_election.cc:302] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4720929b44c4560bfdf08107e137362; no voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c
I20240501 20:11:08.772529 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeba0c03d83a433d9a519ba9fe29b53a" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.772645 15267 raft_consensus.cc:2726] T c1211cfd40674438a14a49818863b3f0 P b4720929b44c4560bfdf08107e137362 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 20:11:08.772712 12874 raft_consensus.cc:3037] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.773175 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeba0c03d83a433d9a519ba9fe29b53a" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.773470 13137 raft_consensus.cc:3037] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.775381 12874 raft_consensus.cc:2454] T eeba0c03d83a433d9a519ba9fe29b53a P 77bf705599644326b6e71b5caf32fc7c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 7.
I20240501 20:11:08.775852 12942 leader_election.cc:302] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:08.777235 13137 raft_consensus.cc:2396] T eeba0c03d83a433d9a519ba9fe29b53a P bb1be87a7aee48a4b405b9d209c61231 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 20:11:08.777601 15267 raft_consensus.cc:2781] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 20:11:08.777850 15267 raft_consensus.cc:686] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [term 7 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:08.780294 13002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e6a39cb39f7c4559b97ffa749a658e7b"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:08.780627 13002 raft_consensus.cc:482] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.780776 13002 raft_consensus.cc:3037] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.784912 13002 raft_consensus.cc:504] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.786527 15267 consensus_queue.cc:237] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.786859 12875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a39cb39f7c4559b97ffa749a658e7b" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.787132 12875 raft_consensus.cc:3037] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.788053 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a39cb39f7c4559b97ffa749a658e7b" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.788393 13137 raft_consensus.cc:3037] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.789988 12875 raft_consensus.cc:2454] T e6a39cb39f7c4559b97ffa749a658e7b P 77bf705599644326b6e71b5caf32fc7c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 7.
I20240501 20:11:08.787375 13002 leader_election.cc:288] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 7 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:08.790416 12942 leader_election.cc:302] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:08.790844 15268 raft_consensus.cc:2781] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 20:11:08.791015 15268 raft_consensus.cc:686] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [term 7 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:08.791307 15268 consensus_queue.cc:237] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.794152 13002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a8b101e9e38b4048bc86d76f97201534"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:08.794370 13002 raft_consensus.cc:482] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.794497 13002 raft_consensus.cc:3037] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.795192 13137 raft_consensus.cc:2396] T e6a39cb39f7c4559b97ffa749a658e7b P bb1be87a7aee48a4b405b9d209c61231 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 20:11:08.798475 13002 raft_consensus.cc:504] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.800806 13002 leader_election.cc:288] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 7 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:08.800722 12875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b101e9e38b4048bc86d76f97201534" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.800967 12875 raft_consensus.cc:3037] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.801301 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8b101e9e38b4048bc86d76f97201534" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.801811 13137 raft_consensus.cc:3037] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.801775 13002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ddb529dd5534f8bb66562e8467c1f12"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:08.801976 13002 raft_consensus.cc:482] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.802084 13002 raft_consensus.cc:3037] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.803835 12875 raft_consensus.cc:2454] T a8b101e9e38b4048bc86d76f97201534 P 77bf705599644326b6e71b5caf32fc7c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 7.
I20240501 20:11:08.804299 12942 leader_election.cc:302] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:08.805049 13137 raft_consensus.cc:2396] T a8b101e9e38b4048bc86d76f97201534 P bb1be87a7aee48a4b405b9d209c61231 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 20:11:08.805239 15267 raft_consensus.cc:2781] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 20:11:08.805310 13002 raft_consensus.cc:504] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.805580 15267 raft_consensus.cc:686] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [term 7 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:08.805930 15267 consensus_queue.cc:237] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.806435 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ddb529dd5534f8bb66562e8467c1f12" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.806703 12746 raft_consensus.cc:3032] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:11:08.806808 13002 leader_election.cc:288] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 6 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:08.806838 12746 raft_consensus.cc:729] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.807104 12746 consensus_queue.cc:260] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.807412 12746 raft_consensus.cc:3037] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.807739 13002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7c3933feed404a4baee16428d2aafacf"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:08.808017 13002 raft_consensus.cc:482] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.808279 13002 raft_consensus.cc:3037] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.810814 12746 raft_consensus.cc:2396] T 4ddb529dd5534f8bb66562e8467c1f12 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.811591 12875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ddb529dd5534f8bb66562e8467c1f12" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.811830 12875 raft_consensus.cc:3037] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.812443 13002 raft_consensus.cc:504] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.813844 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c3933feed404a4baee16428d2aafacf" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.814064 12874 raft_consensus.cc:3037] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.814059 13002 leader_election.cc:288] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 7 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:08.814199 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c3933feed404a4baee16428d2aafacf" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.814574 13137 raft_consensus.cc:3037] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.815173 13002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "436330ea0516443799906257122b38af"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:08.815368 13002 raft_consensus.cc:482] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.815474 13002 raft_consensus.cc:3037] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.816808 12874 raft_consensus.cc:2454] T 7c3933feed404a4baee16428d2aafacf P 77bf705599644326b6e71b5caf32fc7c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 7.
I20240501 20:11:08.817267 12942 leader_election.cc:302] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters:
I20240501 20:11:08.817570 15268 raft_consensus.cc:2781] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 20:11:08.817734 15268 raft_consensus.cc:686] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [term 7 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:08.817986 15268 consensus_queue.cc:237] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.818500 13137 raft_consensus.cc:2396] T 7c3933feed404a4baee16428d2aafacf P bb1be87a7aee48a4b405b9d209c61231 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 20:11:08.818393 13002 raft_consensus.cc:504] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.819270 13002 leader_election.cc:288] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 7 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:08.819926 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436330ea0516443799906257122b38af" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.820214 13137 raft_consensus.cc:3037] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.819976 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436330ea0516443799906257122b38af" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.820422 12746 raft_consensus.cc:3032] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 20:11:08.820545 12746 raft_consensus.cc:729] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.820849 12746 consensus_queue.cc:260] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.821175 12746 raft_consensus.cc:3037] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.823449 13137 raft_consensus.cc:2454] T 436330ea0516443799906257122b38af P bb1be87a7aee48a4b405b9d209c61231 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 7.
I20240501 20:11:08.823879 12942 leader_election.cc:302] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:08.824083 12746 raft_consensus.cc:2396] T 436330ea0516443799906257122b38af P 4e9d799e45f8407590cb3096f5625c62 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 20:11:08.824288 15267 raft_consensus.cc:2781] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 20:11:08.824498 13002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dafd65d56bb243c78cb4490d7da7b18a"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:08.824581 15267 raft_consensus.cc:686] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [term 7 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:08.824697 13002 raft_consensus.cc:482] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.824851 13002 raft_consensus.cc:3037] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.825093 15267 consensus_queue.cc:237] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.806146 12599 catalog_manager.cc:5484] T eeba0c03d83a433d9a519ba9fe29b53a P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 5 to 7, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 7 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:08.827916 13002 raft_consensus.cc:504] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.828841 13002 leader_election.cc:288] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 7 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:08.831070 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dafd65d56bb243c78cb4490d7da7b18a" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.831694 12746 raft_consensus.cc:3032] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 20:11:08.831751 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dafd65d56bb243c78cb4490d7da7b18a" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.831868 12746 raft_consensus.cc:729] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.832065 13137 raft_consensus.cc:3037] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.832216 13002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "17276a219da74d95919c8f427c9920c5"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:08.832182 12746 consensus_queue.cc:260] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.832422 13002 raft_consensus.cc:482] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.832504 12746 raft_consensus.cc:3037] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.832527 13002 raft_consensus.cc:3037] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.835464 13137 raft_consensus.cc:2454] T dafd65d56bb243c78cb4490d7da7b18a P bb1be87a7aee48a4b405b9d209c61231 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 7.
I20240501 20:11:08.835927 12942 leader_election.cc:302] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:08.836275 15269 raft_consensus.cc:2781] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 20:11:08.836151 13002 raft_consensus.cc:504] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.836401 15269 raft_consensus.cc:686] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [term 7 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:08.836669 15269 consensus_queue.cc:237] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.837049 13002 leader_election.cc:288] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 7 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:08.837567 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17276a219da74d95919c8f427c9920c5" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.837747 12745 raft_consensus.cc:3032] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 20:11:08.837843 12745 raft_consensus.cc:729] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.838117 12745 consensus_queue.cc:260] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.838408 12745 raft_consensus.cc:3037] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.839145 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17276a219da74d95919c8f427c9920c5" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.839370 12874 raft_consensus.cc:3037] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.842346 12874 raft_consensus.cc:2396] T 17276a219da74d95919c8f427c9920c5 P 77bf705599644326b6e71b5caf32fc7c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:08.843348 13002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "47b8c200bb05478db26c0d10b25a514c"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:08.843676 13002 raft_consensus.cc:482] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.843904 13002 raft_consensus.cc:3037] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.848284 12745 raft_consensus.cc:2396] T 17276a219da74d95919c8f427c9920c5 P 4e9d799e45f8407590cb3096f5625c62 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:08.848845 12942 leader_election.cc:302] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4720929b44c4560bfdf08107e137362; no voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c
I20240501 20:11:08.848930 13002 raft_consensus.cc:504] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.849169 15269 raft_consensus.cc:2726] T 17276a219da74d95919c8f427c9920c5 P b4720929b44c4560bfdf08107e137362 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 20:11:08.852754 13002 leader_election.cc:288] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 7 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:08.855571 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b8c200bb05478db26c0d10b25a514c" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.856004 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47b8c200bb05478db26c0d10b25a514c" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.856288 12745 raft_consensus.cc:3032] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 20:11:08.856323 12874 raft_consensus.cc:3037] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.856456 12745 raft_consensus.cc:729] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.856786 12745 consensus_queue.cc:260] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.857168 12745 raft_consensus.cc:3037] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.859711 12875 raft_consensus.cc:2396] T 4ddb529dd5534f8bb66562e8467c1f12 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.860532 12942 leader_election.cc:302] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4720929b44c4560bfdf08107e137362; no voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c
I20240501 20:11:08.860945 15269 raft_consensus.cc:2726] T 4ddb529dd5534f8bb66562e8467c1f12 P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 20:11:08.861290 12746 raft_consensus.cc:2396] T dafd65d56bb243c78cb4490d7da7b18a P 4e9d799e45f8407590cb3096f5625c62 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 20:11:08.863890 12745 raft_consensus.cc:2396] T 47b8c200bb05478db26c0d10b25a514c P 4e9d799e45f8407590cb3096f5625c62 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:08.865913 13002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b96f18923f9d41878cb01202e77be033"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:08.866204 13002 raft_consensus.cc:482] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.866343 13002 raft_consensus.cc:3037] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.869141 12874 raft_consensus.cc:2396] T 47b8c200bb05478db26c0d10b25a514c P 77bf705599644326b6e71b5caf32fc7c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:08.869683 12942 leader_election.cc:302] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4720929b44c4560bfdf08107e137362; no voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c
I20240501 20:11:08.870033 15269 raft_consensus.cc:2726] T 47b8c200bb05478db26c0d10b25a514c P b4720929b44c4560bfdf08107e137362 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 20:11:08.874989 13002 raft_consensus.cc:504] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.876549 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96f18923f9d41878cb01202e77be033" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.876816 13137 raft_consensus.cc:3037] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.877735 13002 leader_election.cc:288] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 6 election: Requested vote from peers bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:08.879365 13002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4488bf7917574118915a9355219528ee"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:08.879568 13002 raft_consensus.cc:482] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.879647 13137 raft_consensus.cc:2454] T b96f18923f9d41878cb01202e77be033 P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 6.
I20240501 20:11:08.879695 13002 raft_consensus.cc:3037] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.880062 12942 leader_election.cc:302] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4720929b44c4560bfdf08107e137362, bb1be87a7aee48a4b405b9d209c61231; no voters:
I20240501 20:11:08.880398 15267 raft_consensus.cc:2781] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:11:08.877852 12875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96f18923f9d41878cb01202e77be033" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.880712 15267 raft_consensus.cc:686] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [term 6 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:08.880720 12875 raft_consensus.cc:3032] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:11:08.880934 12875 raft_consensus.cc:729] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 77bf705599644326b6e71b5caf32fc7c, State: Running, Role: LEADER
I20240501 20:11:08.881026 15267 consensus_queue.cc:237] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.884093 12599 catalog_manager.cc:5484] T a8b101e9e38b4048bc86d76f97201534 P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 5 to 7, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 7 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.884558 13002 raft_consensus.cc:504] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.885326 12599 catalog_manager.cc:5484] T 7c3933feed404a4baee16428d2aafacf P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 5 to 7, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 7 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.886132 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4488bf7917574118915a9355219528ee" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.886339 12745 raft_consensus.cc:3032] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:11:08.886333 13002 leader_election.cc:288] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 6 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:08.886426 12745 raft_consensus.cc:729] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.886709 12745 consensus_queue.cc:260] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } }
I20240501 20:11:08.887009 12745 raft_consensus.cc:3037] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.889906 12745 raft_consensus.cc:2396] T 4488bf7917574118915a9355219528ee P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.891165 12874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4488bf7917574118915a9355219528ee" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.891443 12874 raft_consensus.cc:3037] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.891906 12599 catalog_manager.cc:5484] T dafd65d56bb243c78cb4490d7da7b18a P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 6 to 7, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 7 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } }
I20240501 20:11:08.892539 12599 catalog_manager.cc:5484] T e6a39cb39f7c4559b97ffa749a658e7b P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 5 to 7, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 7 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:08.893126 12599 catalog_manager.cc:5484] T 436330ea0516443799906257122b38af P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 6 to 7, leader changed from 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 7 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:08.895615 12874 raft_consensus.cc:2396] T 4488bf7917574118915a9355219528ee P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.896083 13002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "50561f1e0c5b47a78abb1fba84e725fa"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:08.896593 12942 leader_election.cc:302] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4720929b44c4560bfdf08107e137362; no voters: 4e9d799e45f8407590cb3096f5625c62, 77bf705599644326b6e71b5caf32fc7c
I20240501 20:11:08.896926 15268 raft_consensus.cc:2726] T 4488bf7917574118915a9355219528ee P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 20:11:08.897534 13002 raft_consensus.cc:482] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.897691 13002 raft_consensus.cc:3037] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.898670 12875 consensus_queue.cc:260] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.899106 12875 raft_consensus.cc:3037] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.911064 12875 raft_consensus.cc:2396] T b96f18923f9d41878cb01202e77be033 P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.912434 13002 raft_consensus.cc:504] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.913463 13002 leader_election.cc:288] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 7 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:08.914147 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50561f1e0c5b47a78abb1fba84e725fa" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.914436 13137 raft_consensus.cc:3037] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.914393 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50561f1e0c5b47a78abb1fba84e725fa" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.914709 12745 raft_consensus.cc:3032] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 20:11:08.914839 12745 raft_consensus.cc:729] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.915210 12745 consensus_queue.cc:260] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.915611 12745 raft_consensus.cc:3037] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.918440 13137 raft_consensus.cc:2396] T 50561f1e0c5b47a78abb1fba84e725fa P bb1be87a7aee48a4b405b9d209c61231 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:08.920423 13002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1d207517fea64fe1a83a2409157ae2bd"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:08.920711 13002 raft_consensus.cc:482] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.920884 12745 raft_consensus.cc:2396] T 50561f1e0c5b47a78abb1fba84e725fa P 4e9d799e45f8407590cb3096f5625c62 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 20:11:08.920903 13002 raft_consensus.cc:3037] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.921411 12942 leader_election.cc:302] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b4720929b44c4560bfdf08107e137362; no voters: 4e9d799e45f8407590cb3096f5625c62, bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:08.921749 15268 raft_consensus.cc:2726] T 50561f1e0c5b47a78abb1fba84e725fa P b4720929b44c4560bfdf08107e137362 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 20:11:08.925220 13002 raft_consensus.cc:504] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.928009 12599 catalog_manager.cc:5484] T b96f18923f9d41878cb01202e77be033 P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 5 to 6, leader changed from 77bf705599644326b6e71b5caf32fc7c (127.29.219.66) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 6 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:08.935323 12875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d207517fea64fe1a83a2409157ae2bd" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.935597 12875 raft_consensus.cc:3037] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.935943 13002 leader_election.cc:288] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 6 election: Requested vote from peers 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:08.936026 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d207517fea64fe1a83a2409157ae2bd" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.936262 13137 raft_consensus.cc:3037] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.940155 13137 raft_consensus.cc:2396] T 1d207517fea64fe1a83a2409157ae2bd P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240501 20:11:08.940847 13002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b6ba30e296834fb28bf4a88141c24826"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:08.941125 13002 raft_consensus.cc:482] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.941287 13002 raft_consensus.cc:3037] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.942224 12875 raft_consensus.cc:2454] T 1d207517fea64fe1a83a2409157ae2bd P 77bf705599644326b6e71b5caf32fc7c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4720929b44c4560bfdf08107e137362 in term 6.
I20240501 20:11:08.943702 12942 leader_election.cc:302] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77bf705599644326b6e71b5caf32fc7c, b4720929b44c4560bfdf08107e137362; no voters: bb1be87a7aee48a4b405b9d209c61231
I20240501 20:11:08.944051 15267 raft_consensus.cc:2781] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 20:11:08.944201 15267 raft_consensus.cc:686] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [term 6 LEADER]: Becoming Leader. State: Replica: b4720929b44c4560bfdf08107e137362, State: Running, Role: LEADER
I20240501 20:11:08.944240 13002 raft_consensus.cc:504] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.944555 15267 consensus_queue.cc:237] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.945739 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ba30e296834fb28bf4a88141c24826" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.945950 12745 raft_consensus.cc:3032] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 20:11:08.946062 12745 raft_consensus.cc:729] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.946475 12745 consensus_queue.cc:260] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } }
I20240501 20:11:08.946872 12745 raft_consensus.cc:3037] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.950395 12745 raft_consensus.cc:2396] T b6ba30e296834fb28bf4a88141c24826 P 4e9d799e45f8407590cb3096f5625c62 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b4720929b44c4560bfdf08107e137362 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 20:11:08.953207 13002 leader_election.cc:288] T b6ba30e296834fb28bf4a88141c24826 P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 6 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), 77bf705599644326b6e71b5caf32fc7c (127.29.219.66:32867)
I20240501 20:11:08.954092 12875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ba30e296834fb28bf4a88141c24826" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "77bf705599644326b6e71b5caf32fc7c"
I20240501 20:11:08.954954 13002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "70cd215b4f9445499e14c9fd18f7f7fe"
dest_uuid: "b4720929b44c4560bfdf08107e137362"
from {username='slave'} at 127.0.0.1:52368
I20240501 20:11:08.955155 13002 raft_consensus.cc:482] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 20:11:08.955287 13002 raft_consensus.cc:3037] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.955814 12875 raft_consensus.cc:3037] T b6ba30e296834fb28bf4a88141c24826 P 77bf705599644326b6e71b5caf32fc7c [term 5 FOLLOWER]: Advancing to term 6
I20240501 20:11:08.955953 15049 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 20:11:08.956712 12599 catalog_manager.cc:5484] T 1d207517fea64fe1a83a2409157ae2bd P b4720929b44c4560bfdf08107e137362 reported cstate change: term changed from 4 to 6, leader changed from bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68) to b4720929b44c4560bfdf08107e137362 (127.29.219.67). New cstate: current_term: 6 leader_uuid: "b4720929b44c4560bfdf08107e137362" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77bf705599644326b6e71b5caf32fc7c" member_type: VOTER last_known_addr { host: "127.29.219.66" port: 32867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } health_report { overall_health: HEALTHY } } }
I20240501 20:11:08.958115 13002 raft_consensus.cc:504] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68" port: 37951 } }
I20240501 20:11:08.958889 13002 leader_election.cc:288] T 70cd215b4f9445499e14c9fd18f7f7fe P b4720929b44c4560bfdf08107e137362 [CANDIDATE]: Term 7 election: Requested vote from peers 4e9d799e45f8407590cb3096f5625c62 (127.29.219.65:38629), bb1be87a7aee48a4b405b9d209c61231 (127.29.219.68:37951)
I20240501 20:11:08.960032 12745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70cd215b4f9445499e14c9fd18f7f7fe" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "4e9d799e45f8407590cb3096f5625c62"
I20240501 20:11:08.960115 13137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70cd215b4f9445499e14c9fd18f7f7fe" candidate_uuid: "b4720929b44c4560bfdf08107e137362" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "bb1be87a7aee48a4b405b9d209c61231"
I20240501 20:11:08.960291 12745 raft_consensus.cc:3032] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 20:11:08.960317 13137 raft_consensus.cc:3037] T 70cd215b4f9445499e14c9fd18f7f7fe P bb1be87a7aee48a4b405b9d209c61231 [term 6 FOLLOWER]: Advancing to term 7
I20240501 20:11:08.960402 12745 raft_consensus.cc:729] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4e9d799e45f8407590cb3096f5625c62, State: Running, Role: LEADER
I20240501 20:11:08.960660 12745 consensus_queue.cc:260] T 70cd215b4f9445499e14c9fd18f7f7fe P 4e9d799e45f8407590cb3096f5625c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e9d799e45f8407590cb3096f5625c62" member_type: VOTER last_known_addr { host: "127.29.219.65" port: 38629 } } peers { permanent_uuid: "b4720929b44c4560bfdf08107e137362" member_type: VOTER last_known_addr { host: "127.29.219.67" port: 42701 } } peers { permanent_uuid: "bb1be87a7aee48a4b405b9d209c61231" member_type: VOTER last_known_addr { host: "127.29.219.68"